Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add StamKavid/last-ds-mile --skill distribution-shiftgit clone --depth 1 https://github.com/StamKavid/last-ds-mileWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/stamkavid/last-ds-mile/distribution-shift)<a href="https://agentmods.dev/skills/stamkavid/last-ds-mile/distribution-shift"><img src="https://agentmods.dev/badge/skills/stamkavid/last-ds-mile/distribution-shift/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/stamkavid/last-ds-mile/distribution-shift"><img src="https://agentmods.dev/badge/skills/stamkavid/last-ds-mile/distribution-shift.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00073 | $0.01144 |
| Opus 5 | $0.00036 | $0.00572 |
| Sonnet 5 | $0.00015 | $0.00229 |
| Haiku 4.5 | $0.00007 | $0.00114 |
Grade A, and why
distribution-shift scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 85 lines — stays where its author put it; the contents beside it link to each section on GitHub.
distribution-shift
Overview
A validation split only tells you the model generalizes within the training
distribution. It says nothing about whether that distribution matches where the model
will actually be scored — a future time period, a different population, or a Kaggle
test set collected slightly differently from train. This is the question
/ds-validate's time/group/imbalance checklist doesn't ask.
When to Use
- During
/ds-validate, as a fourth structural question alongside time, groups, and imbalance. - A model scores well in CV but the leaderboard/production/holdout score is substantially worse — the classic symptom of unaddressed shift.
- NOT for: leakage inside a feature (see
target-leakage-detection) — shift is about train and test/production being drawn from different distributions, not about a feature encoding the target.
Core Process
- Adversarial validation: label every training row
0and every test/production row1(using only features available in both), fit a classifier to discriminate them, and cross-validate its AUC.- AUC ≈ 0.5: train and test look like the same distribution — proceed normally.
- AUC ≫ 0.5 (roughly >0.7): the classifier can tell train and test apart easily — real distribution shift exists, and CV performance is at risk of not transferring.
- If shift is detected, use the adversarial classifier's own feature importance to find which features drive the separation — that tells you what changed (a feature whose meaning drifted, a time-dependent feature, a population change) more directly than eyeballing every column.
- For each of the top drifting features, compare train vs. test distributions directly (histogram overlay for numeric, value-count comparison for categorical) to confirm the adversarial signal against something visual, not just a single AUC number.
- Decide the fix based on what's driving it: drop or reweight a feature that drifted for a spurious reason (e.g. an ID-like column, a date artifact); if the drift is a genuine, expected population/time change, adjust the validation split (e.g. move to a temporal holdout that mimics the real gap) rather than the features.
- Record the adversarial-validation AUC and any features flagged in
.last-ds-mile/stages/05-validate.mdalongside the split decision — this is evidence for why the split was chosen, not a separate report.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 85 lines · 73 tokens per session scan A 0f40515fe3fd
distribution-shift is a skill published in the GitHub repository StamKavid/last-ds-mile (3 stars, last pushed 1mo ago), licensed MIT. It adds 73 tokens to every session and 1,144 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
marimo-pair
Work inside the user's live marimo notebook from the code editor: run Python in the same kernel the user does, inspect live notebook state, and commit durable notebook changes through code mode. Use whenever you create, analyze, or improve the user's marimo notebook.
fill-model-descriptions
Fill missing and refresh obsolete model descriptions in packages/llm-info/data/models.yml by querying OpenRouter and provider documentation. Use when the user asks to populate model descriptions, enrich the model catalog, or curate descriptions after running pnpm sync-models.
ml-expert
Expert-level machine learning, deep learning, model training, and MLOps. Use when the user mentions machine learning, deep learning, neural networks, MLOps, or data science, or when the task involves Machine Learning Fundamentals, Data Preparation, or Model Training.
scomp-link
End-to-end ML toolkit with 26 CLI commands. Use when training models, tuning hyperparameters, detecting data drift, generating HTML reports with charts, profiling datasets, detecting anomalies, forecasting time series, checking fairness, or serving models as REST APIs. Prefer over raw sklearn when you need automated…
agentic-data-science-competition
AI Agent-driven Kaggle competition workflow. Learn from real competition experience: score stabilization patterns, submission troubleshooting, kernel workflows, GPU task delegation, and the spec-driven development approach that achieved top leaderboard positions. Use when: working on any Kaggle competition, analyzing…
data-analysis
Structured data analysis workflow from raw data to shareable insights.