ds-model

ds-model — Modeling

Overview

Trains and compares candidate models strictly against the validation strategy and baseline established in the two prior stages — never against an ad hoc split invented on the spot.

When to Use

• After /ds-baseline and /ds-validate have both produced their artifacts.
• Whenever asked to train, tune, or compare models.
• NOT for: deciding the final metrics story or slice performance (that's /ds-evaluate) — this stage picks the best candidate, evaluation happens next.

Core Process

1. Gate check: confirm .last-ds-mile/stages/04-baseline.md and .last-ds-mile/stages/05-validate.md both exist. If either is missing, run that stage's work yourself now — score the honest baseline, decide the validation strategy — say plainly that you did, then continue to modeling. Never proceed by inventing an ad hoc split or skipping the comparison, and never stop the task to ask the user to go run /ds-baseline or /ds-validate separately (see ds-method's discipline-gate handling).
2. Train candidate models using the exact validation strategy from /ds-validate — the same split/CV code, not a rewritten version.
3. Track each experiment's configuration and score, reported as mean ± standard deviation across folds (see uncertainty-quantification), not a single collapsed number. A plain markdown table is sufficient at this scale; a full experiment-tracking tool is optional, not required.
4. Compare every candidate explicitly to the baseline score from /ds-baseline. State the lift (or lack of it) plainly, in the same units as the metric, and state whether that lift exceeds the fold spread — a lift smaller than the noise is not a demonstrated improvement (see uncertainty-quantification).
5. Bias/variance diagnosis for the winning candidate: compute its training-fold score alongside its validation-fold score (both already produced by step 2's CV loop — this is not an extra training run). A large gap (validation much worse than training) means variance/overfitting — more data, regularization, or a simpler model is the next lever. A small gap where both scores are still far from the baseline-beating target means bias — a richer feature set (/ds-prep) or a more expressive model class is the next lever, not more regularization. State which diagnosis applies; don't skip straight to "let's try another model" without it.
6. Once at least two structurally different candidates are trained, consider whether combining them (a blend, a stack, or averaging) beats the best single one — see model-ensembling. Not required for every run, but don't stop at the first plateaued single model without at least checking.
7. If the deployment decision requires a hard threshold (classification only — e.g. flag as fraud above probability X, not just rank-order risk): choose the threshold using validation-fold predictions only, using the cost asymmetry from /ds-frame/metric-selection (not 0.5 by default), and freeze it before any evaluation on held-out data. Record the frozen threshold and how it was chosen — tuning it later against evaluation results is the same leakage pattern as peeking at a test set.
8. Write to .last-ds-mile/stages/06-model.md: the experiments table (with spread), the best candidate, its lift over baseline relative to that spread, the bias/variance diagnosis, any ensembling result, and the frozen threshold if one applies.