replay-proven-fix

replay-proven-fix

The operator's standing requirement for a behaviour change: a replayed test case with the input, today's output, the problem, and the output after the fix. This skill is that loop, plus the traps found while running it (homelab, 2026-09-30 → 10-02).

repro-first covers "one bug, one failing test". This skill is for changes to decisions over many inputs, where the risk is what else moves.

The loop

1. Find it in real data, with counts. "10,764 of 11,700 rows repeat", not "lots of noise". Query the real system; never estimate.
2. Freeze the evidence into a committed corpus: snapshots, payloads, and the external state the decision reads (e.g. GitHub issue states). Scrub secrets and user ids. History moves on; the corpus keeps the replay reproducible.
3. Write the fix. Prefer a deterministic rule over a prompt or a threshold tweak.
4. Write the replay: old code vs new code over the corpus.
   • The old code comes from git at the commit before the fix. Never re-implement "how it used to work" by hand.
     • Python: scripts/new_replay.py scaffolds this.
     • Any language: eval "$(scripts/baseline_worktree.sh <BASE_REV>)" checks the old code out into $BASELINE_DIR; run the replay there and in the working tree.
   • First check the baseline reproduces the recorded history. If it can't, the replay isn't faithful, and nothing it says about the fix counts.
   • Write explicit pass criteria and exit non-zero on failure. Include the invariant ("nothing outside the bug changes"), not only the headline ("the 6 alerts became 1").
   • Stub external writes; serve reads from the corpus. A replay never sends anything.
5. Mutation-check the replay. Break the fix in the plausible ways it could be wrong, and confirm the replay fails each time:

   python3 ~/.claude/skills/replay-proven-fix/scripts/mutate.py \
     --file path/to/module.py --find '<exact fix text>' --replace '<broken version>' \
     --precheck 'python3 -m py_compile path/to/module.py' -- <replay command>
   

   • Exit codes: 0 = killed (good), 1 = survived (the replay proves nothing; add the missing check), 2 = setup or invalid mutant. The file is always restored, verified by sha256.
   • Read WHY it failed. A mutant can fail for the wrong reason, e.g. a syntax error your edit introduced; --precheck catches that.
6. Read the changed cases, not just the totals, and judge them against explicit links, not titles or names. For example:
   • a duplicate link (GitHub GraphQL ClosedEvent.duplicateOf);
   • the stored signal or row behind the case;
   • an id that may have been reused.
7. Gate it: add the replay to the project's deploy or CI gate list, plus focused unit tests. Run every gate.
8. Commit with the numbers: input, before, after, the mutants and their results. If a later finding contradicts what an earlier commit message claimed, say so in the next commit message. Push and deploy only with the operator's go.
9. Verify live. "The replay passed" is not "it works in production". After deploying, exercise the real path: one real input pushed through by hand, or the real decision run read-only against live state with writes intercepted. If a deploy step loaded its own script before the pull (bash reads it first), run any newly added gate by hand once.
10. Document: update the project's design guide (why) and runbook (how), and list what is still not covered.