compliance-catalog

Compliance Catalog

Convert a compliance document PDF into a validated OSCAL Catalog JSON through an iterative loop between a deterministic extraction script (generate.py) and a comprehensive validator (validate.py).

┌─ Catalog Agent (loop control, CONFIG edits, gap judgments) ────────────┐
│  ┌─ Deterministic Extraction Pipeline (scripts/) ────────────────────┐ │
│  │ PDF → text extract → structure parse → OSCAL emit                 │ │
│  │       → postprocess → validate (17 rules + trestle)               │ │
│  └───────────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘

  generate.py ──▶ catalog.json ──▶ validate.py
      ▲                                │
      └── FIX_GUIDANCE tells you which ─┘
          function of generate.py to edit

What is an OSCAL Catalog, and what counts as a control

An OSCAL Catalog is NIST's machine-readable representation of a compliance document. OSCAL defines a control as "a requirement or guideline, which when implemented will reduce an aspect of risk related to an information system and its information" — not simply "a numbered thing in the document." A catalog groups controls under group (the document's own Parts / Chapters / Schedules), and each control's textual content lives in a part named statement.

This matters because compliance PDFs almost always contain numbered units that are not controls under this definition: definitions sections, purely administrative articles (commencement dates, amending formulas, short titles), and similar non-normative text. The final catalog.json you deliver must not contain these units — only genuine requirements. See "Extraction is requirement-agnostic; exclusion is a separate, reviewable step" below for the two-pass mechanism (excluded_units.json) that achieves this without weakening the extraction itself.

What Phase 2's FIRST generate.py run does, unaffected by the paragraph above: extract every numbered unit in the PDF as a control, exactly as this document already instructs — extraction stays mechanical and complete. The removal of non-requirement units happens in a later, separate step (Phase 2's second generate.py run, after excluded_units.json is written). The OSCAL definition above explains why the catalog is organized as controls-under-groups and why statement is the right place for extracted prose — it does not change what the first pass extracts.