document-processing

Document Processing

Extract, convert, and structure content from PDFs, images, and other document formats. Handles OCR, text extraction, markdown conversion, email extraction, and structured data output.

When to Use

• Converting scanned documents to searchable text
• Extracting text from PDFs (native or scanned)
• Converting documents to markdown for further processing
• Extracting emails, addresses, or other structured data from documents
• Batch processing document collections

Processing Pipeline

Step 1: Identify Document Type

Determine the processing approach:

Detection:

# Check if PDF has text content
pdftotext input.pdf - | head -20
# If output is empty or garbled, it's a scanned PDF -> use OCR

Step 2: Extract Content

Native PDF Extraction

import pymupdf

doc = pymupdf.open("input.pdf")
for page in doc:
    text = page.get_text("markdown")  # or "text", "html"
    print(text)

OCR with Mistral (for scanned documents)

Requires MISTRAL_API_KEY environment variable. Falls back to tesseract for offline processing if unavailable.

import base64
import httpx

def ocr_page(image_path: str, api_key: str) -> str:
    with open(image_path, "rb") as f:
        image_data = base64.b64encode(f.read()).decode()

    response = httpx.post(
        "https://api.mistral.ai/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "model": "mistral-ocr-latest",
            "messages": [{
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}},
                    {"type": "text", "text": "Extract all text from this image. Preserve formatting, tables, and structure. Output as markdown."}
                ]
            }]
        }
    )
    return response.json()["choices"][0]["message"]["content"]