systematic-troubleshooting

Systematic Troubleshooting

Overview

Guessing at explanations wastes time and corrupts the scientific record. Post-hoc rationalization is indistinguishable from cargo-cult science.

Core principle: ALWAYS investigate root cause before proposing explanations or re-running experiments. Rationalization without investigation is failure.

Violating the letter of this process is violating the spirit of scientific rigor.

The Iron Law

NO EXPLANATIONS WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose why an experiment failed.

When to Use

Use for ANY unexpected experimental result:

• Model performance far below or above expectation
• Analysis output that contradicts prior findings
• Metric values that don't make domain sense (e.g., correlation > 1, negative variance)
• Convergence failures, NaN/Inf in outputs
• Results that change between runs with no code change
• Findings that contradict established literature

Use this ESPECIALLY when:

• Under deadline pressure (temptation to re-run and hope is highest)
• "I know what went wrong" feels obvious
• You've already tried re-running multiple times
• A reviewer asked why results differ from a prior submission
• You don't fully understand the pipeline end to end

Don't skip when:

• The issue seems minor (small discrepancies have real causes)
• You're in a hurry (systematic is faster than rationalized thrashing)
• The paper deadline is close (wrong results submitted is worse than delayed submission)

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Investigate

BEFORE proposing any explanation or re-running:

1. Read Error Messages and Output Carefully
   • Don't skip past warnings — in scientific pipelines, warnings are often the finding
   • Read log files completely, not just the final summary line
   • Note exact metric values, not approximations
   • Record environment: library versions, hardware, OS, date