reckoning

reckoning

!checklist init ${CLAUDE_SKILL_DIR} --force

The runs are done; the numbers are on disk. reckoning is the lens you hold over the results — the step where data becomes settled claims, an explanation of why, and figures a reviewer reads in three seconds. It is the fifth skill of the inquiry suite: it owns step five of doing computational research — analyzing the results until every claim has a verdict and the win has a proven cause. It audits (and guides you to run) a gated pipeline, and it will not advance past a GATE until the checklist tool clears it. That gate enforces order — each step done before the next — not the substance of the work inside it; the tool structures the discipline, it does not audit it, so the rigor is yours to supply.

The one mental shift everything hangs on — analysis is audit, attribution, and the language of evidence, not "compute a mean and see who is bigger". Three things separate this from arithmetic. First, you audit before you read: a too-good number, an outlier, anything surprising is a bug or an infeasible solution until it survives that suspicion — never a breakthrough on first sight. Second, you prove why the method wins with mechanism probes — checkable predictions that are not the main result itself — because winning the table only shows the method works, not that it works for the reason you claim; this is the dividing line between a publishable paper and a strong one. Third, you defend against the garden of forking paths — the danger the agent era amplifies, because when experiments are cheap, unconscious cheating is cheap, and every path you might wander down "passes its tests". The arc is: audit the data → read the distribution and slices → interpret the ablation → probe the mechanism → draw the failure boundary → settle every claim → red-team and render.

The agent is the means, not the oracle. Here it has three jobs: the auditor (run the mechanical pre-read checklist and report), the probe runner (build the counterfactual instances, sweep the structure knob, batch the specification multiverse), and the adversarial reviewer (attack your claims and figures from separated roles). It is not the source of truth and not the judge of what counts. Its gradient points at pleasing you — left soft it confirms your story, and its "review" decays into universally-true platitudes ("the experiments could be more thorough"). So the red-team is run as a system (separated roles, fresh sessions, real-review calibration), and the mechanism question is yours, because an agent will happily call a coincidence a cause.

What you cannot delegate — three judgments. The pipeline parallelizes the labor, but three points stay yours: (1) what the unit of analysis is — instance, not instance×seed; get this wrong and a tenfold-inflated n crushes every p-value into false significance; (2) what prediction the mechanism implies — the probe only has force if you derive a consequence that is not the win itself; (3) which claims to settle how — supported, revised, or dropped, including the honesty to conditionalize or delete a claim that confirmation shrank. Outsource these and you have automated a confident, well-formatted wrong conclusion.

What "done" looks like — the paper already exists in pieces, not "I made some plots". Analysis is over when: every claim has a verdict and the settling is recorded; the mechanism explanation has at least two probes behind it; the failure boundary is characterized; and every figure is script-generated and has survived the red-team. At that point the paper exists — the claims are its argument, the protocol is its experiments section, the probes are its discussion, the difference table is its related work. If you have those, stop analyzing. "I made some plots and the method wins" is not the terminus.