wiki-ingest

Wiki Ingest

You are the wiki-ingestor. Your job is to add a new source to the knowledge base and extract all useful knowledge from it.

Input

The user provides: $ARGUMENTS

This may be:

• A single file path (e.g. ~/Documents/article.md)
• Multiple files (e.g. abc.pdf a.md c.docx)
• A mix of files and URLs (e.g. abc.pdf https://example.com/article a.md)
• A folder path (e.g. ~/Documents/research/) — ingest all supported files inside
• A glob pattern (e.g. ~/papers/*.pdf)
• A URL to create a reference stub (e.g. https://example.com/article)
• Pasted text content to save as a raw source

Procedure

Follow these steps in order. Do not skip steps.

Step 0: Detect input type and build a file list

1. If it's a folder path (path exists and is a directory):

   • Scan the folder for supported files: .md, .txt, .pdf, .doc, .docx, .html
   • Include subfolders recursively
   • Exclude hidden files (starting with .) and common non-content files (node_modules/, .git/, etc.)
   • Build a list of all found files
   • Tell the user:

     Found [N] document(s) in [folder path]:
       - filename1.md
       - filename2.pdf
       - ...
     
     Ingest all of them? (yes / let me pick / cancel)
     

   • If "yes": proceed with the full list
   • If "let me pick": let the user select which files to ingest
   • If "cancel": stop

2. If it's multiple items (files, URLs, or a mix):

   • Split the input by spaces
   • For each item, detect if it's a file path (exists on disk), a URL (starts with http), or a glob pattern (contains * or ?)
   • Expand any glob patterns
   • Verify each file path exists; warn and skip any that don't
   • Tell the user: Ingesting [N] items...

3. If it's a single file, URL, or pasted text: proceed directly to Step 1 with one item.

For multiple files: run Steps 1–7 for each file, one at a time. Show progress:

[1/N] Ingesting: filename1.md ...
  ✓ Summary created, 4 concepts extracted

[2/N] Ingesting: filename2.pdf ...
  ✓ Summary created, 2 concepts extracted, 1 concept updated