debugging-pipelines

Debugging Data & ML Pipelines

Systematic root cause investigation for broken data pipelines, unexpected model behavior, and analysis anomalies.

Iron Law: NO FIXES WITHOUT ROOT CAUSE INVESTIGATION

Checklist

1. Reproduce the problem — get a minimal reproducible example
2. Characterize the symptom — what exactly is wrong? Exact numbers.
3. Form hypotheses — list 3-5 possible causes, ranked by likelihood
4. Test hypotheses — binary elimination, most likely first
5. Identify root cause — confirmed with evidence, not assumption
6. Fix at the root — address root cause, not symptom
7. Verify fix — run the failing case to confirm resolution
8. Check for recurrence — add validation or test to prevent regression

Problem Types and Investigation Paths

Type 1: Data Pipeline Error (Exception/Crash)

Error → Read full traceback → Identify failing line
         ↓
         Check input data at the failing step
         ↓
         Is input shape/type what code expects? → NO → data preparation bug
         ↓ YES
         Is a column missing? → check upstream step or schema change
         ↓
         Is a value out of range? → check upstream validation

# Minimal reproduction
import traceback

try:
    result = pipeline.fit_transform(problematic_df)
except Exception as e:
    traceback.print_exc()
    print(f"\nInput shape: {problematic_df.shape}")
    print(f"Input dtypes:\n{problematic_df.dtypes}")
    print(f"Null counts:\n{problematic_df.isnull().sum()}")

Type 2: Model Performance Degradation

Performance dropped from training to production? Work through this list:

1. Data distribution shift?
   → Compare feature distributions: training vs current
   → KS test or PSI (Population Stability Index)

2. Target distribution shift?
   → Compare class ratios or target mean

3. Missing features?
   → Check if all expected features are present and non-null

4. Feature engineering bug?
   → Compare sample features manually (pick 3 rows, trace by hand)

5. Wrong model version loaded?
   → Log model hash, confirm version

6. Data leakage in training (inflated training metrics)?
   → Re-evaluate on a fresh hold-out never used during development