diagnose

diagnose — failures into actionable side information

A scalar reward says how much a candidate failed; it does not say why, and "why" is the only thing an editor can act on. Where RL back-propagates a scalar into weights, natural-language feedback back-propagates into prompt/tool/skill edits — and the richer it is, the larger the update extractable from a handful of rollouts.

What it produces

{
  "split": "val", "tag": "cand_003",
  "reflective_dataset": [
    {"task_id": "t12", "Inputs": "<what the task asked>",
     "Generated Outputs": "<what the agent produced>",
     "Feedback": "<the scorer's diagnosis>",
     "Trajectory": "<path to this task's full trace>"}
  ],
  "clusters": [
    {"signature": "confirm write", "tasks": ["t12", "t19"], "score_lost": 1.6,
     "tag": "BEHAVIORAL", "blast_radius": ["t3", "t7"]}
  ],
  "kept_good": ["t1", "t4"]
}

scripts/run.py emits everything except tag (one of KNOWLEDGE, BEHAVIORAL, DECISION / PERMISSION, CAPABILITY-GAP) and blast_radius, which it leaves null because they need judgement — filling them in is the work below. kept_good is the set the gate's no-regression check protects.

What counts as a failure

Not only zero-score tasks. Three kinds are real lost score and routinely missed: partial credit (scored e.g. 0.5 because one part of the action was wrong), communication / omission (the action happened but the required information was never reported or confirmed), and near-miss (≈0.7–0.9, one small correct change from a pass — the cheapest marginal gain per edit, easiest to overlook while staring at the zeros).

Separate always-failing (mean ≈ 0 — a root-cause fix) from flaky (0 < mean < 1 — a consistency fix; find what the passing trials do and make it reliable). The reward is the honest signal: a per-task Feedback line comes from the last trial and can disagree with the graded mean.

Where the trace comes from

The rollout record supplies the score and the feedback; the trajectory supplies the failure site, and the site is half of the cluster key. The runner owns the trace format, so never assume one — the location is asked for, not guessed: