systematic-debugging

Systematic Debugging

Do root-cause investigation before proposing fixes.

Iron Law

No fix without evidence for root cause.

This is non-negotiable. Every fix must trace back to a proven root cause. A fix applied without understanding why it works is not a fix — it is a mask.

Four Phases

Phase 0: Check Known Issues

• If known-issues.md exists at the project root, search it for the error message, error code, or failing test name.
• If a match is found, try the documented solution first. If it resolves the issue, stop — no further investigation needed.
• If no match or the documented solution doesn't work, proceed to Phase 1.

Phase 1: Investigate

• Read the full error output — not just the last line.
• Reproduce the bug reliably. If you cannot reproduce, you cannot fix. For tests that fail only in certain orderings (test pollution), run find-polluter.sh from this skill's directory to identify which test is corrupting shared state.
• Check recent changes — what changed since it last worked?
  • If context-snapshot.json exists at the project root: run git rev-parse HEAD and compare to git_hash in the file.
    • Hashes match (fresh): the changed_files and recent_commits fields answer this for the committed changes.
    • Hashes differ (stale): the snapshot is from a previous commit; do not use it. Run the two commands of the next line.
  • If absent: run git log --oneline -10 and git diff HEAD~1..HEAD --name-only.
• Add instrumentation (logging, breakpoints) at component boundaries.
• Multi-component systems: When the system has multiple components (CI → build → signing, API → service → database), add diagnostic logging at EACH component boundary before proposing fixes:

  For EACH component boundary:
    - Log what data enters the component
    - Log what data exits the component
    - Verify environment/config propagation
    - Check state at each layer
  
  Run once to gather evidence showing WHERE it breaks
  THEN analyze evidence to identify the failing component
  THEN investigate that specific component
  

  Example (multi-layer system):

  # Layer 1: Workflow
  echo "=== Secrets available in workflow: ==="
  echo "IDENTITY: $([ -n "${IDENTITY:+x}" ] && echo SET || echo UNSET)"
  
  # Layer 2: Build script
  echo "=== Env vars in build script: ==="
  [ -n "${IDENTITY:+x}" ] && echo "IDENTITY in environment" || echo "IDENTITY not in environment"
  
  # Layer 3: Signing script
  echo "=== Keychain state: ==="
  security list-keychains
  security find-identity -v
  
  # Layer 4: Actual signing
  codesign --sign "$IDENTITY" --verbose=4 "$APP"
  

  This reveals which layer fails (e.g., secrets → workflow OK, workflow → build FAILS).
• Trace data/control flow backward from the error to its source.