01-pdf-preprocessor

Agent 1: PDF Preprocessor

Role

Converts a PDF academic paper into clean Markdown text suitable for downstream LLM processing. This is the entry point of the entire pipeline.

Position in Pipeline

• Phase: 1 (Preprocessing)
• Depends on: User-provided PDF file
• Produces for: A2 (Asset Extractor), A3 (Commitment Builder), A4 (Discourse Parser), A9 (Image Filter)
• LLM Required: No

Inputs

Input	Source	Format
PDF file path	User input	File path string

Execution Steps

1. Extract text using Docling:

   from docling.document_converter import DocumentConverter
   converter = DocumentConverter()
   conv_res = converter.convert(pdf_path)
   md = conv_res.document.export_to_markdown()
   

   • Docling handles multi-column academic layouts, preserves heading hierarchy, and formats tables as Markdown
   • This is the same Docling instance used by A2 for figure extraction — reuse the converter result if both agents run together

2. Clean the extracted text:

   • Remove HTML comments: <!-- ... -->
   • Remove excessive whitespace while preserving paragraph breaks
   • Remove OCR noise from figure regions (garbled text from images)
   • Normalize unicode characters
   • Keep mathematical notation in plain text form where possible

3. Quality check:

   • If extracted text is less than 500 characters, the extraction likely failed
   • In that case, try an alternative extraction method or ask the user

4. Save output:

   • Write cleaned markdown to workspace/markdown.md

Output Schema

Plain text Markdown file with:

• Paper title as # Title
• Section headings as ## Section Name
• Subsection headings as ### Subsection Name
• Paragraphs separated by double newlines
• Figures/tables referenced inline (but actual extraction is A2's job)

Output Path

• workspace/markdown.md

Validation

• File must be > 500 characters
• Must contain at least 2 section headings (## )
• Should preserve the paper's hierarchical structure