xberg-batch-extraction

Batch extraction

Use this when processing a directory or glob of documents in one pass. xberg batch shares one extraction config across every file, runs extractions concurrently, and returns a JSON envelope — failures on individual files do not abort the run.

Basic usage

# Glob expands to many paths; results come back in a JSON envelope (default)
xberg batch *.pdf

# Mixed formats, markdown content for LLM ingestion
xberg batch docs/*.docx --content-format markdown

# Recurse with the shell, then extract
xberg batch $(find ./corpus -name '*.pdf')

batch defaults to --format json (vs --format text for single extract). The envelope contains results, total_ms, per_file_ms and, when failures occur, errors. Each results entry is a full extracted document. One input may produce multiple documents, so do not pair results with inputs by position. per_file_ms is aligned with inputs; inspect errors for per-input failures.

xberg batch reports/*.pdf \
  | jq '.results[] | {chars: (.content | length), mime: .mime_type}'

Parallelism

--max-concurrent caps how many files extract at once. When omitted, the scheduler derives document concurrency from the total thread budget. Lower it on memory-constrained hosts or when OCR/ML models are active, since each in-flight extraction holds its own buffers. Layout-heavy batches are further limited (1 concurrent extraction for all-PDF-layout batches, 2 for mixed layout):

# Cap at 4 concurrent extractions
xberg batch scans/*.pdf --ocr true --max-concurrent 4

--max-threads additionally caps total internal threads (Rayon, ONNX intra-op, the batch semaphore) for tightly constrained environments:

xberg batch *.pdf --max-concurrent 2 --max-threads 4

Per-file config overrides

A single shared config does not always fit. --file-configs points at a JSON file mapping each path to its own override object, merged on top of the shared config for that file only: