papyrus: Instructions file for Codex

AGENTS.md

Instructions for AI coding agents working on Papyrus. Read this before changing anything. CLAUDE.md points here; this file is the source of truth.

What Papyrus is

A universal document ingestion engine. Any file in, agent-ready Markdown out. It is a deterministic library first, with a CLI, an HTTP API and a demo web app layered on top.

The one rule

bytes → detect → parser → Document IR → renderer → markdown
                              ↓
                          chunks.jsonl

Parsers never emit Markdown. Renderers never parse files. Everything meets at the Document IR (src/papyrus/ir.py). If you find yourself writing a # or a | inside src/papyrus/parsers/, stop — that string belongs in src/papyrus/renderers/markdown.py.

This is what makes a new format a single self-contained file. Breaking it costs the project its main property, so it is not a stylistic preference.

Core principles

1. Degrade, don't crash. Messy input produces a Document with a warning appended. Only genuinely unreadable input raises ParseError. A parser that raises something unexpected is wrapped by Converter and surfaces as ParseError — never a bare traceback.
2. Never trust a filename. detect() reads magic bytes first. A PDF named invoice.docx is a PDF.
3. No LLM in the conversion path. Conversion is deterministic and free. Same bytes in, same Markdown out. An AI cleanup pass may exist later, but it is opt-in and it sits after the IR, never inside a parser.
4. Use the format's own signal before a heuristic. Word marks header rows with w:tblHeader; HTML has <th>; PyMuPDF reports the header it detected; csv.Sniffer exists. Guessing from cell contents is the fallback, not the first move — see split_header(rows, hint).
5. Preserve document order. python-docx hands you paragraphs and tables as separate collections; python-pptx hands you shapes in z-order; PDF text and tables come from two different passes. All three are reconstructed. A table rendered under the wrong heading is a correctness bug, not a formatting nit.
6. Every uploaded file is hostile. Never execute input, never shell out, never write it outside a temp directory, always sanitise the filename with safe_name(). Every ceiling lives in Limits.
7. Preserve provenance. sha256, source format, page anchors. A converted document must be traceable to the bytes it came from.