progressive-verification-debugging

Progressive Verification Methodology

Problem

When debugging complex systems (ML training, distributed systems, integrations), it's easy to:

• Waste time checking logs without a strategy
• Assume the wrong component is broken
• Try complex solutions before understanding root cause
• Get overwhelmed by multiple potential failure points

Context / Trigger Conditions

Use this methodology when:

• Silent crashes with no error messages
• "Works on my machine" but fails elsewhere
• Multiple layers: environment → code → data → model
• Version compatibility issues suspected
• Need to isolate environment vs code issues

Classic symptoms:

• Process exits silently after starting
• GPU utilization 0-1% despite training script running
• Empty log files despite process being active
• Different behavior across platforms (Mac vs WSL2 vs Linux)

Solution

Core Principle

"先要用简单代码验证环境，然后判断是不是配置问题"

Always start with the simplest test, gradually increase complexity.

Progressive Verification Steps

Create a test ladder from simplest to most complex:

Level 1: Basic Infrastructure
├── Test: torch.randn(2, 3).cuda()
└── Validates: CUDA driver, GPU access

Level 2: Core Components
├── Test: nn.Linear(10, 5).cuda()
└── Validates: Model creation, CUDA memory allocation

Level 3: Real Models
├── Test: ResNet50(weights=None).cuda()
└── Validates: Complex model architecture

Level 4: Data Pipeline (Synthetic)
├── Test: DataLoader with random tensors
└── Validates: DataLoader, multiprocessing

Level 5: Data Pipeline (Real)
├── Test: PIL.Image.open + transforms
└── Validates: File I/O, image decoding

Level 6: Training Loop
├── Test: Single forward + backward pass
└── Validates: Optimization, gradient flow

Decision Tree

Does Level N pass?
├── Yes: Proceed to Level N+1
└── No: Issue is at Level N
    └── Fix before proceeding

Example Test Script

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torchvision.models import resnet50
from PIL import Image

print("=== Progressive Verification ===")

# Level 1: Basic CUDA
print("Level 1: Basic CUDA")
x = torch.randn(10, 10).cuda()
print("  PASSED")

# Level 2: Model on CUDA
print("Level 2: Model on CUDA")
model = nn.Linear(10, 5).cuda()
y = model(x)
print("  PASSED")

# Level 3: ResNet50 on CUDA
print("Level 3: ResNet50 on CUDA")
resnet = resnet50(weights=None).cuda()
print("  PASSED")

# Level 4: DataLoader (synthetic)
print("Level 4: DataLoader (synthetic)")
class DummyDataset(Dataset):
    def __len__(self): return 10
    def __getitem__(self, i): return torch.randn(3, 224, 224), 0

ds = DummyDataset()
loader = DataLoader(ds, batch_size=4, num_workers=0)
for x, y in loader:
    x, y = x.cuda(), y.cuda()
    output = resnet(x)
    break
print("  PASSED")

# Level 5: Real data loading
print("Level 5: Real data loading")
# Test actual file loading here
print("  PASSED")

print("All levels passed!")