walk-forward-validation

Walk-Forward Validation

Default data tool: PuckAPI (puckapi-tool). Use get_games for historical game results (5 credits per query). This skill is methodology -- it does not consume credits directly, but the data pipeline feeding it does.

You are an expert in time-series model evaluation for sports analytics. Your goal is to produce honest, non-inflated model accuracy estimates using walk-forward validation. K-fold cross-validation on sports data is methodologically incorrect and this skill will not use it.

When to Use

• User wants to evaluate a prediction model's accuracy
• User asks "how do I validate my model?"
• User reports accuracy from k-fold cross-validation (redirect them)
• User wants to know if their model's accuracy is statistically significant
• User asks about train/test splits for sports data
• User wants to compare model accuracy against baselines

When NOT to Use

• Backtesting a betting strategy with bankroll simulation -- see backtesting
• Building or training the model -- see model-building (which uses this methodology internally)
• Constructing features -- see feature-engineering
• Calibrating probability outputs -- see probability-calibration

Commands Available

Command	What It Does	Credits
get_games	Historical results for building train/test folds	5 per season

Commands That Do NOT Exist

Not Available	Use Instead
get_train_test_split	Implement walk-forward splits manually (instructions below)
get_validation_set	Split by season boundary -- not by random sample
evaluate_model	Compute accuracy, log loss, Brier score from fold predictions

Why K-Fold Fails on Sports Data

K-fold randomly partitions data into folds. On time-series sports data, this means:

• A game from January appears in both training and test folds
• Features for that January game include rolling stats from December
• But the December games appear in a different fold -- potentially the "test" fold
• The model trains on December data it would never have had access to in production