report

report — did it work, and by how much?

The result of a run is not "we made edits" — it is a defensible answer to did this actually work, and by how much. report lays three numbers side by side: where the seed started (val), where the best candidate landed (val), and the single held-out test number that counts. It is what a human reads to decide whether to ship.

Runs standalone as /cap-evolve:report, or headlessly as the last step of cap-evolve run — same scripts/run.py either way. report is a phase SCRIPT, not a cap-evolve subcommand; invoke the script.

How to read the three numbers

The honest reading is always test vs baseline, with val as a sanity check in between. scripts/run.py produces the numbers; this is the judgment you add on top:

• test ≈ baseline → no real gain. The val improvement was overfitting or noise the gate let through. Do not ship; tighten gate_k_se or add trials.
• test ≫ baseline → genuine improvement on data the optimizer never saw. Ship.
• val ≫ test → the classic overfit signature: the optimizer learned the val set, not the capability. The reported val→test gap is the overfitting, quantified.
• pass^k far below pass^1 → the gain is fragile across trials; the agent sometimes succeeds but not reliably. A high mean with low pass^k is not a dependable win (τ-bench's point).

Every number is rendered with its stderr when one was measured, because "0.71" and "0.71 ± 0.08" support very different decisions. A gain smaller than the noise floor is not a result — say so plainly rather than quoting the point estimate alone.

Output contract

scripts/run.py owns both artifacts and writes them deterministically from the run dir — do not hand-write or paraphrase them, or two runs stop being comparable.

report.md is exactly this skeleton (bracketed lines appear only when they apply):

# cap-evolve run report — <run_id>

[> **NOT FINALIZED** — no held-out test number. Run the finalize phase first; …]
[> **No holdout** (train == val == test). The test number below is a *fit* metric, …]

- Best candidate: `<best_id>`
- Baseline val: <r> ± <se>
- Best val: <r> ± <se>
- **Held-out test (optimized skills): <r> ± <se>**  (pass^1=…, pass^k=…)
[- Held-out test (baseline `<baseline_id>` skills): <r> ± <se>]
[- **Test improvement (optimized − baseline): <+Δ>**]
[- Val→test gap: <+Δ> — selection optimism on val; this gap IS the overfitting]
- Iterations: <n>
[- Optimized for: <consuming model> (tier <t>)]

[<sealed note — omitted entirely when the run was never finalized>]