post-run-review

/post-run-review — structured Assess → Codify

Ports the AIDE submit_review pattern (MIT, aide/agent.py L19–44 + parse_exec_result L296–339). Ours adds the C4 failure-class classification — bug | bad-hyperparam | bad-hypothesis — which no other autonomous-research harness currently cleanly separates.

When to invoke

Automatically, as the final step of the Assess phase, after any experiment with a journal node (journal.py add … was called at Plan time). Don't invoke on trivial scripts.

What the skill does

Given:

• Journal node id (--node N)
• Log / stdout from the run (--log PATH)
• Optional metric override (--metric NAME=VAL)

The agent reads the log, forms a structured verdict, and writes:

{
  "is_buggy": 0 | 1,
  "failure_class": null | "bug" | "bad-hyperparam" | "bad-hypothesis",
  "metric_name": "val_loss",
  "metric_value": 3.42,
  "lower_is_better": 1,
  "summary": "one-paragraph what-happened",
  "learn_block": "<optional [LEARN] emitted separately>",
  "dead_end_block": "<optional [DEAD-END] emitted separately>"
}

and then:

1. Calls journal.py close <node> --metric ... --is-buggy ... --failure-class ... --analysis "...".
2. If is_buggy=1 and it's a clear durable rule: emits a [LEARN] block in its response (Stop hook captures it).
3. If failure_class=bad-hypothesis: emits a [DEAD-END] block.
4. If actual_delta for this run was predicted: calibration.py close <run_id> "<hypothesis>" <actual>.

Failure-class meanings (IMPORTANT)

class	means	action
bug	implementation error (shape mismatch, off-by-one, NaN from wrong init)	fix the code, retry; emit [LEARN] with the gotcha
bad-hyperparam	logic is sound but config misses (lr too high, batch wrong, seed bad)	tune; keep the hypothesis alive
bad-hypothesis	idea itself doesn't work at the scale tested	emit [DEAD-END]; do NOT re-propose this direction

Mis-classifying a bad-hypothesis as a bug will cause the agent to loop on a fundamentally broken idea. Mis-classifying a bug as a bad-hypothesis will kill directions that would have worked after a fix. Be honest. When uncertain, prefer bad-hyperparam — it keeps the option open and triggers the cheapest next action (tune).