ingest-pipeline

Wiki Ingest Pipeline — Plan & Optimize Procedures (reference)

The Step 1.4 topic-tree plan + confirmation gate, the Step 3 optimize (destructive restructure) procedure, and the final-report template for claude-wiki-pages-ingest-agent. The agent reads this skill at the start of Step 1.4 and Step 3, and when composing the final report. Schema authority remains vault/CLAUDE.md; the contract, preflight, page-write steps, auto-heal delegation, synthesis, model selection, and hard rules live in the agent body. Every page the pipeline produces follows the house voice — see skills/voice: a one-line plain-language definition first, then typed content in the engineer register, no marketing inside the vault.

Step 1.4 — Plan the topic tree (externalize, then confirm)

The topic-tree shape is the most consequential decision of the run. Errors here cascade into every page's parent: and path:, and into the Obsidian graph structure. Externalize the plan so the user can review or edit before any page is written.

PDF sources (source_format: pdf): when a source is a PDF under raw/assets/, apply the PDF ingest path documented in skills/ingest/SKILL.md — "PDF sources (I4): source_format: pdf" — before planning the topic tree. The required fields (source_format: pdf, attachment_path, extracted_at) must appear in the source note, and the PDF remains immutable in raw/assets/ throughout. Classification and dedup proceed identically to text sources.

1.4a — Write the plan

Write to vault/output/_pipeline-plan-YYYY-MM-DD.md (git-ignored; no frontmatter required). Structure:

# Ingest plan — YYYY-MM-DD

## Sources in this run
- <source-1.md> — N entities, M concepts
- <source-2.md> — N entities, M concepts
...

## Entities and concepts extracted

> For each extracted item, apply the **classification checklist** in
> `skills/ingest/SKILL.md` — assign exactly one `type` (and, for entities,
> one `entity_type`) drawn from `ontology-profile-v1` in `vault/CLAUDE.md`.
> Never invent out-of-enum values; flag ambiguous cases for human review.
>
> Also apply the **two-pass alias-aware dedup** from `skills/ingest/SKILL.md`
> before marking any item as `new`. Pass 1 = exact title match; pass 2 =
> alias-aware match against existing pages' `aliases` fields. Mark items as
> `existing` if either pass matches, and plan an additive extension (never a
> duplicate). See "Dedup: two-pass existence check (I2)" in
> `skills/ingest/SKILL.md` for the full procedure.

- [<new|existing>] <Entity/Concept name> — `type: <type>` (entity: `entity_type: <value>`) — from <source(s)>
...

## Proposed topic tree

wiki/<existing-or-new-topic>/
├── <topic>.md                   [new | existing]   # folder note (legacy _index.md if already present)
├── <page>.md                    [new | update]
├── <subtopic>/                  [new | existing]
│   ├── <subtopic>.md            [new | existing]   # folder note
│   └── <page>.md                [new | update]
...

## Folder size check
- <topic>/: N direct children (target ≤ 12)
- <topic>/<subtopic>/: N direct children (target ≤ 12)

## Graph color groups needed
- <new-top-level-topic> → next palette color
- (or: none)

## Open decisions
- <any ambiguities the model resolved — e.g., "placed X under Y instead of Z because …">