systematic-debugging

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues and usually force a second debugging cycle.

Core principle: ALWAYS find root cause before attempting a permanent fix. Symptom fixes are failure.

Debugging report quality is part of the job. A report that lists guesses without evidence is not a passing debugging result.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

Phase 1 always applies. How much of Phases 2-4 apply, and what the final report requires, is set by Severity Triage and Scope & Mode below — read both before investigating.

Diagnostic Instrumentation Exemption

Adding temporary code to collect evidence is NOT a fix. Permitted during Phase 1: print/log statements, temporary breakpoints or debug flags, probe scripts, temporary test harnesses, system commands (df -h, lsof, strace, tcpdump) to observe runtime state. "Permitted" here is about scope (it's investigation, not a fix), not about pre-approval — tcpdump and env specifically are not pre-approved in allowed-tools and require asking first; see references/safety-and-authorization.md.

Rules: mark diagnostic code clearly (// DEBUG-INVESTIGATION or # DIAG), remove it after root cause is identified, and it must not change program behavior — only observe.

When to Use

Use for any technical issue: test failures, bugs in production, unexpected behavior, performance problems, build failures, integration issues, flaky tests, race conditions, Configuration drift, third-party breakage.

Use this ESPECIALLY when: under time pressure, "just one quick fix" seems obvious, you've already tried multiple fixes, a previous fix didn't work, or you don't fully understand the issue.

Severity Triage (Do This First)

+----------+---------------------+------------------------------+------------------+
| Severity | Characteristics     | Investigation                | Time Budget      |
+----------+---------------------+------------------------------+------------------+
| P0       | Production down,    | Mitigate first (P0 Protocol),| Mitigate: <15min |
| Critical | data loss, revenue  | root cause after stable      | Root cause: async|
|          | impact, security    |                               |                  |
+----------+---------------------+------------------------------+------------------+
| P1       | Feature broken,     | Full 4-phase process,        | 30-60min         |
| High     | blocking users,     | no shortcuts                 |                  |
|          | test suite failing  |                               |                  |
+----------+---------------------+------------------------------+------------------+
| P2       | Minor bug, cosmetic | Phase 1 required; Phase 2    | 15-30min         |
| Medium   | edge case, non-     | skippable if cause is        |                  |
|          | blocking            | obvious; Phase 3 collapses   |                  |
|          |                     | to one hypothesis line       |                  |
+----------+---------------------+------------------------------+------------------+