ingesting-images

Ingesting Images

Visual image classification for a source whose text-axis ingestion is already complete. Produces SUBSTANTIVE entries appended to corpus.commons/{corpus}/sources/converted/IMAGE-INDEX.yaml, deletes DECORATIVE files, updates the source's Pass H verification log count.

Completes the image axis when ingesting-resources ran text-only (or image classification was abandoned mid-run from context budget).

When to use

• The source's IMAGE-INDEX.yaml entries are stub-format or labelled PARTIAL and you want full coverage.
• A source was ingested text-only originally and you want to add the image axis now.
• A source's images were re-extracted (e.g., after a substantive PDF revision) and need re-classification.

When not to use

• Full source ingestion: use ingesting-resources.
• Searching the corpus for an existing source: use matching-references.
• Routine reading of an existing classified source: read corpus.commons/{corpus}/sources/converted/IMAGE-INDEX.yaml directly.

Prerequisites

• The source's deep reference exists at corpus.commons/{corpus}/references/{prefix}-{slug}-deep.md.
• Either (a) the source PDF / EPUB exists at corpus.commons/{corpus}/sources/original/{slug}.{ext} (so images can be re-extracted), or (b) the image directory at corpus.commons/{corpus}/sources/converted/{md-slug}-images/ already contains extracted files.

Procedure

1. Resolve scope. If the argument is a single slug, classify that source. If all, scan IMAGE-INDEX.yaml for entries whose comments include PARTIAL or Description not yet authored, and process each affected source in turn.
2. Ensure images extracted. If {md-slug}-images/ is empty or missing, run python3 scripts/extract-images.py corpus.commons/{corpus}/sources/original/{slug}.{ext} to populate.
3. Read existing index entries for the source from the corpus's IMAGE-INDEX.yaml to know what's already classified. Do not duplicate entries already present.
4. Split into waves. For sources over ~100 images, group the file list into waves of 64 (8 batches × 8 files conceptually). One wave = one read-only Sonnet dispatch + one splice + one commit. This bounds context per dispatch, gives a clean recovery point if a session is interrupted, and lets the operator pause between waves without losing state.
5. Dispatch one read-only Sonnet agent per wave. The agent gets the Read tool only: no Bash, no Edit, no Write. Brief it with: (a) the full list of file paths for this wave, (b) the SUBSTANTIVE / DECORATIVE criteria, (c) the exact YAML-line format for substantive entries, (d) the hard rule that data tables / exercise data sets / source documents are substantive (do not flag as decorative just because they lack diagrams), (e) instruction to read each file and report, never fabricate from filename. The single-agent-of-64 pattern is materially more context-efficient than fanning out 8 parallel agents of 8.
6. Stage the wave's YAML in /tmp. Write the agent's substantive entries to /tmp/{slug}-wave{N}.yaml. This keeps the in-progress wave out of git and recoverable if validation fails.
7. Delete decoratives file-by-file. The auto-mode classifier blocks bulk-glob rm of pre-existing files. Chain individual rm commands with && (one per file). Then verify the new disk count.
8. Splice the staging YAML into IMAGE-INDEX.yaml. Use sed -i.bak to update the PARTIAL header (entries / decoratives deleted / unclassified remaining), then cat /tmp/{slug}-wave{N}.yaml >> IMAGE-INDEX.yaml. Validate with python3 -c "import yaml; yaml.safe_load(open(...))": fails fast on any malformed entry.
9. Commit each wave. Convention: image-ingestion: {slug} wave {N} — N kept, M deleted [corpus-data] with a body summarising the chapter / topic coverage. Per-wave commits are the recovery point; if a later wave goes sideways, the wave-N commit is intact.
10. On the final wave, switch PARTIAL → FINAL in the header, with the reconciliation line indexed + deleted = original-extraction-count. Update the planning log entry from PARTIAL to the consolidated final summary.
11. Append one consolidated line per source to _planning/image-ingestion-log.md (or whichever the operator names). Per-wave entries belong in commit messages, not the log. Drift between log header and actual index entries is a known drift hazard: verify the index count with python3 -c "import yaml; d=yaml.safe_load(open(...)); print(len([e for e in d if e.get('file','').startswith('{slug}-images/')]))" before writing FINAL counts.
12. Update Pass H at corpus.commons/{corpus}/references/_audit/_ingest_pass_H_{prefix}-{slug}_verification.md with the refreshed image-classification counts (substantive / decorative).
13. Verify and report: total images processed, substantive count, decorative count, decorative rate, anomalies, recovery actions needed (e.g., orphaned files on disk without index entries).