memory-corpus-ingest

Corpus Ingest

Bring a large dataset into the assistant's working knowledge without stuffing it into memory. The model is a library: the workspace holds the stacks (the raw files, cold and complete), memory holds the card catalog (a small set of map pages that say what exists, when it is from, and where to look), and a purpose-built retrieval skill is the librarian that walks to the right shelf on demand.

Two invariants drive everything below:

1. Raw data never enters the memory corpus. Nothing from the dataset is written into memory/concepts/ except the map pages, and nothing is ever appended to memory/buffer.md (bulk buffer appends trip the consolidation burst guard and per-run caps; the map bypasses the buffer entirely via assistant memory ingest).
2. The map stays small. Roughly 10 to 50 pages regardless of corpus size. If the corpus doubles, the pages get denser or the slices get coarser; the page count does not double.

Procedure

Step 1: Scope and confirm

Identify the source and its size before committing:

du -sh /path/to/raw-corpus
find /path/to/raw-corpus -type f | wc -l

Tell the user what will happen: the raw files move into the workspace, a bounded number of summarization passes read them once to build the map, the map is ingested into memory, and a lookup skill is authored for drill-in. Skimming a large corpus is real LLM work that costs time and money; confirm before starting. For Fathom recording exports, read references/fathom.md first for format discovery and slicing guidance.

Step 2: Cold-store the raw corpus

Land the raw files under an imports directory in the workspace, one directory per source:

Screen for credentials BEFORE copying: an arbitrary corpus can carry secret material, and anything landed under imports/ becomes reachable by workspace tools, backups, and retrieval flows.

cd "$VELLUM_WORKSPACE_DIR"
# 1a. Screen for secret-bearing FILE NAMES; review every hit with the user.
find /path/to/raw-corpus \( -name '.env*' -o -name '*.key' -o -name '*.pem' \
  -o -name '*credential*' -o -name '*secret*' -o -name 'cookies*' \
  -o -path '*tokens*' -o -path '*oauth*' \) -print

# 1b. Screen file CONTENTS for credential shapes. --hidden and --no-ignore
#     matter: rg skips dotfiles and gitignored paths by default, which is
#     exactly where credentials live. Capture the FULL list (no truncation):
#     every file named here must be excluded below or cleaned with the user
#     before it lands.
rg -l -i --hidden --no-ignore \
  "api[_-]?key|access[_-]?token|refresh[_-]?token|client[_-]?secret|password\s*[=:]|passwd|bearer |AKIA[0-9A-Z]{16}|BEGIN [A-Z ]*PRIVATE KEY" \
  /path/to/raw-corpus > /tmp/corpus-secret-hits.txt
cat /tmp/corpus-secret-hits.txt

# 2. Build rsync exclusions from the content hits (paths relative to the
#    corpus root), then copy with ALL flagged paths excluded.
sed 's|^/path/to/raw-corpus/||' /tmp/corpus-secret-hits.txt > /tmp/corpus-secret-exclusions.txt
mkdir -p imports/<source>
rsync -a --exclude='.env*' --exclude='*.key' --exclude='*.pem' \
  --exclude='tokens/' --exclude='oauth/' --exclude='cookies*' \
  --exclude-from=/tmp/corpus-secret-exclusions.txt \
  /path/to/raw-corpus/ imports/<source>/