systematic-debugging

Systematic Debugging

Replace guess-and-check with an evidence loop that localizes where and why behavior diverges; python-testing implements the fix and incident-response owns live outages.

Workflow

1. Preserve evidence: capture the exact error, relevant stack trace, command, inputs, versions, environment differences, timing, and recent changes before touching anything. Keep large sanitized logs in local artifacts; read by incident time, request ID, or error context, retaining the first causal failure and any truncation limits.
2. Reproduce, reduce, localize: find the shortest reliable reproduction, minimize it while keeping the same failure, and compare each boundary against the nearest known-good comparator.
3. Test one falsifiable hypothesis at a time: state X causes the failure because Y evidence predicts Z observation, change one variable in a reversible fixture or add narrow instrumentation, and discard failed hypotheses instead of layering fixes.
4. Name the root cause: Explain the triggering condition, the faulty assumption or invariant, the propagation path, and why existing controls missed it; never blame timing, the environment, or a third party until that path and the missing resilience are understood.
5. Fix within the requested scope: when the task includes a fix, reuse that authorization, write a regression test for the broken behavior, implement the smallest root-cause correction, and verify the symptom plus affected checks. Broaden to the full gate when repository policy or the change's risk requires it.
6. Report: Return symptom and impact, minimal reproduction, evidence and ruled-out hypotheses, root cause and propagation path, the authorized fix or recommended correction, regression proof, and residual uncertainty with the next probe.

Gotchas

• Diagnosis does not authorize fixes: a request to diagnose authorizes investigation, not implementation; observe read-only, reproduce in an isolated temporary directory, and change product code only when the user also asks for a fix.
• Stop stacking failed fixes: after three failed fix attempts or hypotheses that expose different shared-state failures, stop stacking fixes and reassess the architecture, reproduction, or problem statement. Summarize what the evidence rules out, continue independent safe probes, and ask only when missing information affects scope, correctness, cost, or reversibility.
• Instrument every pipeline boundary once: record presence, shape, identity, status, timestamps, and correlation ids, never secret values; remove the instrumentation unless it has durable value.