cv-lb-gap-acknowledgment

CV-LB Gap Acknowledgment

Problem

A common MLOps trap: trusting CV score as the final metric.

Reality:

• Many experiments show CV improvement but LB degradation
• The gap can be 0.005-0.01 (5-10% of total error) on tabular competitions
• Days of CV optimization may yield worse LB results
• "Best CV" ≠ "Best LB"

The trap: Continue iterating on CV because "the score is improving", missing that CV is overfitting to noise.

Context / Trigger Conditions

Use this skill when:

• OOF score keeps improving but LB score plateaus or drops (red flag!)
• Spending days tuning hyperparameters without LB improvement
• Comparing models on CV only without LB validation
• Trusting cross-validation as the "final" metric for submissions
• Adding more features because CV says so
• More complex model beats simpler one on CV
• Longer training (more iterations) improves CV

Solution: Mandatory 5-Stage Validation Pipeline

Stage 1: CV (Cross-Validation)
  ↓ Compute OOF score
Stage 2: Submission
  ↓ Submit to LB
Stage 3: LB Validation
  ↓ Compare CV vs LB
Stage 4: Analysis
  ↓ Diagnose gap
Stage 5: Decision
  ↓ Iterate / Pivot / Stop

Never skip Stage 2-4! CV is necessary but not sufficient.

Mathematical Reasons for CV-LB Gap

1. Overfitting to OOF Noise

OOF predictions are still predictions on training data (just held out in folds). They are not independent of training process.

Variance decomposition:

Var(OOF) = Var(true_error) + Var(model_selection) + Var(noise)

• OOF score includes model selection variance (you picked this model because OOF was high)
• LB score is out-of-sample (no selection bias)

2. Distribution Shift (Train vs Test)

• Train: Same distribution you optimize on
• Test: May have:
  • Different feature distributions (covariate shift)
  • Different label distributions (prior shift)
  • Different relationships (concept shift)

Example (Spaceship Titanic):

• Train: 8693 rows
• Test: 4277 rows (different distribution)
• CV-LB gap: 0.005-0.01 (0.8124 OOF → 0.8078 LB)