ds-validate

ds-validate — Validation Design

Overview

Decides how the data will be split for honest evaluation, driven by the data's actual structure (time, groups, imbalance) rather than by whatever split is easiest to code.

When to Use

• Before /ds-model — this is a Hard Gate /ds-model checks for.
• Whenever asked to set up train/test splits or cross-validation.
• NOT for: picking which model to try (that's /ds-model) — this stage fixes the split strategy first so it can't later be tuned to flatter a specific model.
• Scope: tabular supervised learning. The TimeSeriesSplit guidance below covers validating a model on time-ordered rows; it is not a forecasting validation stack (no backtest windows, no horizon-aware refitting, no hierarchical reconciliation). If the target is a future value of a series, say so and stop — see README → Scope.

Core Process

1. Ask: is there a time dimension where future data could leak into past predictions? If yes, use a temporal split or backtesting scheme — never shuffled cross-validation.
2. Ask: are there groups (e.g. the same customer or patient across multiple rows) that must not span both train and validation? If yes, use grouped cross-validation (e.g. GroupKFold).
3. Ask: is the target imbalanced? If yes, use stratified splits so folds preserve class balance.
4. Ask: is there a fixed test set (Kaggle-style) or a known deployment population this model will actually be scored against? If yes, run adversarial validation between training data and that population before finalizing the split — see distribution-shift. A split that looks fine internally can still fail to predict real transfer if the test/production distribution differs from training.
5. If none of the above apply, plain (or stratified) k-fold is fine — state that explicitly rather than choosing it by default without checking.
6. Implement with the sklearn splitter that matches the answer — don't hand-roll a split when a splitter class already exists for the case. See the reference table below.
7. Write to .last-ds-mile/stages/05-validate.md: the chosen strategy, why, the distribution-shift check and its result, and the exact split/CV code to be reused identically in /ds-model.