time-series-walk-forward-validation

Time Series Walk-Forward Validation

Context

Random K-fold cross-validation on time series data silently destroys validity. A model can score 0.85 on K-fold and 0.65 on live data, wasting weeks of effort. This skill encodes the validation discipline proven in a retail SKU recommendation project (F1 76.5%, walk-forward stable ±0.2pp over 4 weeks).

The core lesson: time goes one direction, and your validation must respect that. Any information from the future — even an "innocent" aggregate like mean purchase frequency — can leak and inflate your CV score.

Guidance

Step 1: Use Walk-Forward, Never K-fold

# ❌ WRONG: K-fold on time series (future leaks into train)
from sklearn.model_selection import KFold
for train_idx, val_idx in KFold(5).split(X):
    model.fit(X[train_idx], y[train_idx])
    score = model.score(X[val_idx], y[val_idx])
    # X[val_idx] is randomly scattered through time!
    # Train contains rows AFTER validation rows.

# ✅ RIGHT: Walk-forward (each fold uses only past data)
train_weeks = [16, 17, 18, 19]
val_weeks = [20]
model.fit(X[train_weeks], y[train_weeks])
score = model.score(X[val_weeks], y[val_weeks])

# Slide forward
train_weeks = [16, 17, 18, 19, 20]
val_weeks = [21]
# ... at least 4 windows for variance estimate

Step 2: Pre-Train Data Leakage 8-Item Checklist

Before ANY model training, verify:

[ ] Training time range strictly < Evaluation time range
[ ] All aggregate features (co-occurrence, statistics, embeddings)
    computed ONLY on train period
[ ] Candidate set generation rule matches what you'll have at prediction time
[ ] N-formula / threshold / quantile inputs contain NO label information
[ ] Eval set customer set ⊆ train-seen customers
    (cold-start customers evaluated separately)
[ ] No "improvement" can be explained as "future information leaked"
[ ] Walk-forward split has ≥4 evaluation windows (for variance estimate)
[ ] Customer/segment populations are stable across windows
    (no massive churn mid-evaluation)