document-granular-decompose

Document Granular Decompose

Core Goal

• Parse a local document through POST /mineru_with_images.
• Always force return_txt=true.
• Read environment variables for endpoint, request identity, and model routing:
  • UNSTRUCTURED_API_BASE_URL (example: https://your-unstructured-host:7770)
  • UNSTRUCTURED_AUTH_TOKEN
  • UNSTRUCTURED_PROVIDER (optional)
  • UNSTRUCTURED_MODEL (optional)
• Return only plain fulltext (prefer API txt; fallback to joined result[].text).

Triggering Conditions

• Need robust document fulltext extraction for PDF/Office/image files.
• Need image-aware MinerU parsing but only textual output for downstream chunking/search/summarization.
• Need to standardize provider/model/token input via environment variables instead of ad-hoc command parameters.

Workflow

1. Prepare environment variables.

export UNSTRUCTURED_AUTH_TOKEN="your-fastapi-bearer-token"
export UNSTRUCTURED_API_BASE_URL="https://your-unstructured-host:7770"
# Optional routing overrides. Omit them to let the server choose its defaults.
export UNSTRUCTURED_PROVIDER="vllm"
export UNSTRUCTURED_MODEL="Qwen/Qwen3.5-122B-A10B-FP8"

1. Run extraction and print fulltext to stdout.

python3 scripts/mineru_fulltext_extract.py \
  --file "/absolute/path/to/document.pdf"

1. Save fulltext to a local file when needed.

python3 scripts/mineru_fulltext_extract.py \
  --file "/absolute/path/to/document.pdf" \
  --output "/absolute/path/to/fulltext.txt"

Request Contract

• Endpoint resolution:
  • --api-url if provided
  • else UNSTRUCTURED_API_BASE_URL + /mineru_with_images
  • else fail fast with missing environment variable error
• Method: POST multipart form.
• Query params:
  • Force return_txt=true (always set by script).
• Form fields sent:
  • file (required)
  • provider (optional, from UNSTRUCTURED_PROVIDER when set)
  • model (optional, from UNSTRUCTURED_MODEL when set)
• Header sent:
  • Authorization: Bearer $UNSTRUCTURED_AUTH_TOKEN