ingest

ingest

Turn a PDF into a DoclingDocument JSON kept internally (never Markdown) so every extracted element keeps its {page_no, bbox, charspan} provenance for the Phase-8 citation anchor. A pure text-layer quality gate decides native-text-vs-re-OCR before any OCR runs; degraded / handwriting pages are flagged for a human, never silently OCR'd as fact.

This is the suite's document/PDF path. The structured-data (CSV/XLSX) path is dataset-analyze (load_table + data_quality); ingest does not duplicate it.

Two modules, mirroring the suite's pure-core / engine-at-the-edge split (like pii_sweep):

• scripts/ingest_gate.py — the PURE gate (stdlib only; golden-testable with no model): per-page diagnose_page(...) → decide_doc(...) conservative rollup.
• scripts/ingest.py — the Docling edge (the only docling importer; imports it lazily): ingest(pdf_path, *, out_dir, ...) → writes the DoclingDocument JSON and returns an IngestResult.

The verified Docling / RapidOCR / OCRmyPDF facts (the API surface, the coordinate-origin trap, the confidence/nan semantics, the CPU-latency budget) live in references/prior-art.md (the Phase-6 research gate) — consult it before changing a convert call, a backend, or a confidence threshold.

The pipeline

Call one function; it runs the gate and applies the decision.

from scripts.ingest import ingest

result = ingest(
    pdf_path,            # the source PDF
    out_dir=work_dir,    # where the DoclingDocument JSON is written
    deskew=False,        # OCRmyPDF preprocess (Tesseract-gated; see the seam)
)

Internally (design §3):

1. Source identity — SHA-256 the file (source_sha256). Provenance is geometry and artifact identity: a citation ties back to which file.
2. Pass #1, do_ocr=False — one Docling parse over the native text layer. Its per-page native text + parse_score are what the gate sees, and the doc is reused if the decision is native (no second parse).
3. Gate (pure) — diagnose_page labels each page native_ok / image_only / garbled_text / uncertain_review; decide_doc rolls those up conservatively into a doc decision: native / ocr_images / force_full_doc_ocr / review.
4. Apply the decision — native reuses pass #1; ocr_images (do_ocr=True, force_full_page_ocr=False) lets Docling OCR the image regions; force_full_doc_ocr (force_full_page_ocr=True) overrides a present-but-bad text layer; review never silently OCRs. OCR uses RapidOCR (no system binaries).
5. Normalize provenance + persist — force every item's prov bbox to a single coord_origin, then save_as_json (never Markdown).
6. Bates post-pass + degraded flags — capture Bates stamps separately (keeping {page_no, bbox}), and flag degraded / low-confidence / uncertain pages from res.confidence + the gate diagnoses.