lore-ingest

/lore:lore-ingest

Follow the lore skill for all conventions — including its Finding the lore ladder, which resolves $LORE (a path in the user's message, else cwd if it is a lore, else the project's ## Knowledge Base section, else hard fail). Then read $LORE/CLAUDE.md — where it differs from these defaults, it wins.

Default flow is interactive: after each file, surface the key takeaways in the report so the user can steer emphasis before the next one. If the user asks for a batch run, process everything straight through and report once at the end.

1. Find new and changed files

find "$LORE/raw" -type f -printf '%P\n'

(find, not ls — a dropped folder of documents and dotfiles must be seen too; paths are relative to raw/.)

Each found file is in one of three states, decided against log.md (the ledger) and the file's content hash:

• NEW — no ledger entry for this filename. Process per §2.
• PROCESSED — the latest ledger entry for this filename matches the current file's hash (or records no hash, in which case no change can be detected). Skip.
• CHANGED — the latest ledger entry's recorded hash differs from the current file: the raw file was replaced or edited outside the lore flow. Update per §2b.

A ledger entry for a filename is a heading matching ^## \[YYYY-MM-DD\] (ingest|skip) \| <filename>$ — match the whole filename field, anchored at both ends, never a substring: spec.pdf occurs inside the heading for v2_spec.pdf, so a substring test (rg -F "spec.pdf") would classify a newly dropped spec.pdf as already processed and silently never ingest it. The log is append-only, so the last matching heading is the latest. Classify each candidate:

python3 - "$LORE" "<filename>" <<'PY'
import hashlib, re, sys
from pathlib import Path
lore = Path(sys.argv[1]); name = sys.argv[2]
log = (lore / "log.md").read_text(encoding="utf-8")
pat = r'^## \[[0-9-]{10}\] (?:ingest|skip) \| ' + re.escape(name) + r'\s*$'
matches = list(re.finditer(pat, log, re.M))
if not matches:
    print("NEW"); sys.exit()
detail = log[matches[-1].end():].split('\n## ', 1)[0]      # latest entry wins
m = re.search(r'sha256:([0-9a-f]{12})', detail)
cur = hashlib.sha256((lore / "raw" / name).read_bytes()).hexdigest()[:12]
print("CHANGED" if (m and m.group(1) != cur) else "PROCESSED")
PY