uncertainty-quantification

uncertainty-quantification

Overview

A single cross-validated score is a sample from a distribution, not the truth. Two numbers that differ by less than the fold-to-fold noise are the same number wearing different digits. This skill exists so this plugin's own numbers don't commit the exact over-claiming sin the rest of it exists to catch.

When to Use

• Reporting any CV or resampled score in /ds-model or /ds-evaluate.
• Comparing a candidate model's score to the baseline, to another candidate, or to a score from a different validation scheme (e.g. CV vs. a temporal holdout) — the "is this real lift or noise" question.
• NOT for: choosing the split strategy itself (see ds-validate) — this skill quantifies the noise in whatever split was chosen, it doesn't choose the split.

Core Process

1. Never report a single fold's score as "the" score. Report the mean and the standard deviation (or a percentile interval) across folds — cross_val_score already returns per-fold values; use them, don't collapse to .mean() alone.
2. If the dataset is small (roughly under a few thousand rows) or the metric is noisy by nature (e.g. AUC on a rare positive class), repeat the CV with several different random_state seeds and pool the spread across repeats, not just across folds — fold variance alone understates the true uncertainty on small data.
3. Before calling one score "better than," "worse than," or "consistent with" another, compare the gap between them to the spread of each. A gap smaller than the fold standard deviation is not a demonstrated difference — say so explicitly rather than picking the higher number and moving on. Treat this as a screening heuristic, not a test: it is deliberately conservative, and the reason it can't be upgraded into a p-value is in "What fold spread can and cannot tell you" below.
4. State uncertainty in the same units as the metric everywhere it's reported — in the experiments table in /ds-model, and in the final number in /ds-evaluate — not as a caveat added only in one place and dropped elsewhere.
5. For a held-out temporal or sealed check performed once (not part of the CV loop), don't manufacture a fake standard deviation from n=1 — say plainly that it's a single point estimate with no variance, and treat any comparison to the CV mean as directional evidence only, not a statistical test.