rmsle-zero-threshold-asymmetry

RMSLE Zero-Threshold Asymmetry

Problem

When optimizing post-processing for RMSLE metrics, "smarter" adaptive zero-thresholds that use historical min-sales or store-family-level statistics can WORSE leaderboard scores compared to simple fixed thresholds like < 0.1 → 0.

Symptoms

• Model improvements (better features, better CV) appear to have NO effect or NEGATIVE effect on LB after changing post-processing
• Zero ratio in submission increases significantly (e.g., 8% → 12%) after "improving" the zeroing logic
• Items with ~60-70% historical zero rate and ~30-40% non-zero rate are most affected
• Controlled experiment (same model + different post-processing) reveals large LB gap

Root Cause

RMSLE has a fundamental asymmetry:

Predicting small positive when actual = 0:  log1p(0.27)^2 = 0.057  (small error)
Predicting 0 when actual = 2.88:            log1p(2.88)^2 = 1.84   (huge error)

For items with ~68% zero rate and ~32% non-zero rate (mean ~2.9 when non-zero):

• Expected error of predicting 0.27: 0.68 * 0.057 + 0.32 * 1.26 = 0.44
• Expected error of predicting 0: 0.68 * 0 + 0.32 * 1.84 = 0.59

Predicting a small positive value is 25% better than predicting 0, even though the item is zero 68% of the time. The penalty for missing a non-zero sale (log1p) far exceeds the penalty for over-predicting a zero sale.

Solution

Rule 1: Use simple fixed thresholds for RMSLE

# GOOD: Simple, proven threshold
predictions[predictions < 0.1] = 0

# BAD: Complex adaptive threshold — can aggressively zero legitimate predictions
min_threshold = historical_min_nonzero_sales * 0.5
mask = (predictions > 0) & (predictions < min_threshold) & (zero_rate > 0.5)
predictions[mask] = 0  # This zeros 1,172 legitimate predictions!

Rule 2: Verify post-processing changes with controlled experiments

When changing post-processing, create a controlled submission:

1. Take the SAME model predictions
2. Apply ONLY the post-processing change
3. Submit both versions to compare LB impact