seed-reader

seed-reader — Evaluation Report Generation & Metric Interpretation

Render one evaluation run's JSONL logs as a self-contained markdown report, and understand every metric in it. Also covers the gate-scoring companion (analyse_gate) that emits the frozen GO / STOP / ITERATE decision.

When to Use

• After an eval run completes and you need to read its results
• When asked "how did the run go?" or "did hypotree win?"
• When debugging why an arm performed a certain way (duplicates, conflicts, recovery, etc.)
• When you need the formal gate decision (GO / STOP / ITERATE)
• When comparing two runs or diagnosing data-quality issues

How to List Available Run-Ids

Every run's logs live in eval/runs/<run-id>/. The directory names are the run-ids.

# List all run-ids (directories that contain episode logs)
ls eval/runs/

# Check which runs have completed (have a gate_decision.json)
ls eval/runs/*/gate_decision.json 2>/dev/null

# Check a specific run's contents
ls eval/runs/<run-id>/

Run-ids are validated: 1–64 chars of [A-Za-z0-9._-]. Typical convention: YYYY-MM-DD<x> (e.g., 2026-07-28a).

Step 1 — Generate the Report

# Render to stdout
uv run python -m eval.seed_reader --run-id <RUN_ID>

# Write to a file (recommended — the report is long)
uv run python -m eval.seed_reader --run-id <RUN_ID> --output eval/runs/<RUN_ID>/REPORT.md

# Non-default eval directory
uv run python -m eval.seed_reader --run-id <RUN_ID> --eval-dir /path/to/eval/

--run-id is mandatory. Reports are only comparable within a run — mixing two runs' logs would silently invalidate every aggregate. The tool enforces this by checking that each log's own run_id field matches the directory.

Step 2 — Score the Gate Decision

# Emits JSON: {"decision": "GO"|"STOP"|"ITERATE", "criteria": {...}, ...}
uv run python -m eval.analyse_gate eval/ --run-id <RUN_ID>

# Save alongside the report
uv run python -m eval.analyse_gate eval/ --run-id <RUN_ID> | tee eval/runs/<RUN_ID>/gate_decision.json