aminer-exp-extraction

Experiment Data Extraction

One script, one chain: md -> preprocess -> GLM filter -> LLM -> JSON. No manifests, no run state, no monitoring — just extraction. A single model service (Zhipu BigModel, default glm-5.3-flash) powers both the sentence filter and the extraction.

Pre-flight

1. Install requests from requirements.txt.
2. BIGMODEL_API_KEY must be set — it authenticates BOTH model stages (sentence filter + extraction) as Authorization: Bearer (OPENAI_API_KEY accepted as fallback). Never print key values. No other credential or internal service is used.
3. Optional overrides (env or flags):
   • LLM_CHAT_URL — default https://open.bigmodel.cn/api/paas/v4/chat/completions (used by both stages)
   • LLM_MODEL — default glm-5.3-flash (fast variant, used by both stages; glm-5.3 / glm-5.2 also valid)
4. Input is the paper's markdown: a LOCAL file (--md/--md-dir), or paper_id,md_url CSV rows (--csv, md downloaded to --md-cache, cached across re-runs). Local mode: file stem = paper_id.

Run

# single paper
python3 extract_experiments.py --md /path/to/paper.md -o out.json

# batch: one md per paper, named <paper_id>.md
python3 extract_experiments.py --md-dir md_papers/ -o-dir out_json/

# batch from paper_id + md_url (CSV: header paper_id,md_url; md downloaded to md_cache/)
python3 extract_experiments.py --csv papers.csv --md-cache md_cache/ -o-dir out_json/

Per-paper failures don't stop the batch; exit code 2 means at least one failed. CSV downloads are cached — re-runs skip already-downloaded papers.

Output

One JSON per paper: paper_id, paper_title, research_problem(_description/_aliases), domain, experiments[] (name, type, methods, datasets, metrics, key_results, conclusion, limitations, evidence), plus stats (sentence counts, filter backend, elapsed). Schema identical to the production workflow's predictions.

API contract

The skill contacts exactly ONE service: the public Zhipu BigModel chat-completions API. No internal/AMiner gateway is called anywhere (the SciBERT /filter/batch path was removed; stale vendored call sites raise explicitly).