quant-paper-extractor

Quant Paper Extractor

Batch-convert quantitative research report PDFs (量化研究研报) to structured JSONL paper cards. Two-phase pipeline writing into the repo papers library:

papers/raw/{paperId}.pdf
      │
      ▼ Phase 1: PDF → Markdown (pdf_to_markdown.py)
papers/markdown/{paperId}.md
      │
      ▼ Phase 2: Markdown → card (agent-driven extraction)
papers/cards/{paperId}.jsonl
      │
      ▼ Phase 3: refresh derived artifacts
papers/context_brief.md + papers/index.jsonl

Setup

Scripts at scripts/. Run via python scripts/<name>.py from this skill's directory. Dependencies: pip install -e ..

Pre-conditions

The papers directory lives at the repo root (papers/; override with EVOSCIENTIST_PAPERS_DIR). Resolve it once and reuse:

PAPERS=$(python -c "from EvoQuant.papers.paths import resolve_papers_dir; print(resolve_papers_dir())")

Layout (all three layers share the paperId join key):

$PAPERS/raw/       ← PDFs, renamed to {paperId}.pdf (see Red Line 8)
$PAPERS/markdown/  ← auto-created; converted markdown
$PAPERS/cards/     ← auto-created; extracted JSONL cards
$PAPERS/manifest.jsonl  ← processing-state ledger

Legacy layouts are dead. rawpaper/, markdown/, wiki/ in a workspace are deprecated — if you meet them, point the user at python -m EvoQuant.papers.migrate instead of writing there.

markdown/ is auto-created by Phase 1; create the cards dir up front:

mkdir -p "$PAPERS/cards"

Phase 1: PDF → Markdown

Run the conversion script (fully automated, no LLM needed):

python scripts/pdf_to_markdown.py \
  --rawpaper-dir "$PAPERS/raw" \
  --markdown-dir "$PAPERS/markdown" \
  --manifest-path "$PAPERS/manifest.jsonl"

This script:

1. Scans all .pdf files in the raw dir (any filename — the hash is the identity)
2. Computes SHA-256 hash of each PDF's binary content → paperId
3. Incremental skip: if markdown/{paperId}.md already exists, skip
4. Extracts text via three-tier fallback:
   • pymupdf4llm.to_markdown() — native Markdown output (best quality)
   • pymupdf.open() → page.get_text() — plain text with page headers
   • pypdf.PdfReader() → page.extract_text() — last resort
5. Hard-truncates at 120,000 characters
6. Writes structured Markdown to markdown/{paperId}.md
7. Updates manifest.jsonl with status