debug-playbook

Debug Playbook

This is the literal procedure a frontier model follows when debugging. Follow it step by step; the order is the point. Skipping ahead to "the obvious fix" is the single most expensive mistake in debugging.

Operating loop

1. Capture the failure verbatim. Copy the exact error text, stack trace, failing assertion, or wrong output into your notes before touching anything. You will compare against this later — after a change, "a different error" and "the same error" are opposite results, and memory will lie to you about which one you're seeing.
2. Reproduce before reasoning. Run the smallest thing that fails: the one test (<runner> <path>::<test>), the one request (curl the endpoint), the one function (a REPL call). If you cannot reproduce, STOP — collect environment facts (versions, env vars, data state) and report what's missing. Never debug from a description of a failure you haven't seen.
   • Suspected flaky? Run it 10× in a loop and record the failure rate before anything else. A 3/10 failure is a different class of bug (ordering, time, shared state) than a 10/10 failure.
3. Read the trace bottom-up, and find YOUR frame. The deepest frame in code you own is where evidence starts. The reported line is where the error surfaced, not where the bug lives — treat it as the end of a thread to pull, not the answer.
4. Write a hypothesis ledger. List 2–3 candidate causes explicitly (in a comment, scratch file, or your working notes). For each: what it predicts you'd observe, and the cheapest check that would kill it. Rank by (prior probability × cheapness of the check).
5. Run the discriminating experiment. Pick the check that produces a DIFFERENT result under hypothesis A vs hypothesis B — not the one that merely confirms your favorite. Targeted print/log at a boundary, a narrowed input, a bisected commit range (git bisect), or commenting out one layer are all discriminators. One change at a time; revert each probe before the next.
6. State the mechanism sentence. Before writing any fix, you must be able to say: "X causes Y because Z" — where you have OBSERVED X and Y, and Z explains why this exact symptom appears (not merely "could appear"). If you cannot fill in Z, you have a correlation, not a cause. Return to step 4.
7. Fix at the cause site, minimally. The fix belongs where the mechanism sentence says the defect is — not where the error surfaced. If the fix feels large, re-check: you may be redesigning around the bug instead of fixing it.
8. Regression test that fails first. Write the test, run it against the UNFIXED code (stash the fix: git stash, run, git stash pop), confirm it fails with the captured failure — then confirm it passes with the fix. A regression test that never failed proves nothing.
9. Hunt the siblings. The same defect pattern usually exists elsewhere: grep for the same call, the same copy-pasted block, the same misused API. Report siblings even if fixing them is out of scope.
10. Full suite + cleanup. Run the whole test suite, remove every probe and debug print, and re-read your diff — the diff should contain the fix and the test, nothing else.