debugging-and-error-recovery

Debugging and Error Recovery

Overview

The most expensive debugging session is the one that never happens because someone guessed the fix, shipped it, and created three new bugs in the process. Debugging is not intuition — it's a systematic process of elimination. The best engineers aren't the ones who never write bugs; they're the ones who can find and fix any bug in bounded time.

The debugging contract: When something breaks, stop adding features immediately. Preserve evidence. Reproduce the failure. Isolate the minimum trigger. Fix the root cause, not the symptom. Guard against regression. Every skipped step is a debt that compounds.

Real-world impact: Teams that follow structured debugging resolve incidents 3x faster than teams that guess. The "quick fix" that bypasses root-cause analysis returns as a production outage — usually at 3 AM, usually worse than the original problem.

When to Use

• Tests fail after a code change
• The build breaks
• Runtime behavior doesn't match expectations
• A bug report arrives
• An error appears in logs or console
• Something worked before and stopped working

The Stop-the-Line Rule

When anything unexpected happens:

1. STOP adding features or making changes
2. PRESERVE evidence (error output, logs, repro steps)
3. DIAGNOSE using the triage checklist
4. FIX the root cause
5. GUARD against recurrence
6. RESUME only after verification passes

Don't push past a failing test or broken build to work on the next feature. Errors compound. A bug in Step 3 that goes unfixed makes Steps 4-10 wrong.

Iron Rules

These target the debugging failure modes specific to AI agents. Each is absolute — no time pressure overrides them.

1. No fix before a reproduction. A plausible diagnosis is a hypothesis, not a finding. If you can't make the failure happen on demand, you can't know your fix works — you can only hope.
2. Change one variable at a time. If you changed three things and the bug disappeared, you don't have a fix — you have two unnecessary changes and an unverified guess. Revert and isolate.
3. Never suppress the signal. Wrapping in try/catch, increasing a timeout, adding a retry, or loosening a check to make an error disappear is symptom suppression, not a fix. Each of these is only valid when you can state why the underlying cause makes it the correct response.
4. "It works now" without "because" is not fixed. If you can't explain in one sentence why it broke, the bug is still there — you've just stopped seeing it. Prove causality: revert the fix, confirm the failure returns, re-apply.
5. Re-run the original failure, verbatim. "This should fix it" is a prediction. The exit gate is the original failing command rerun with output showing it now passes — plus the full suite for regressions.