Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add topprismdata/cultivating-ml-agent --skill kaggle-oof-lb-validation-protocolgit clone --depth 1 https://github.com/topprismdata/cultivating-ml-agentWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/topprismdata/cultivating-ml-agent/kaggle-oof-lb-validation-protocol)<a href="https://agentmods.dev/skills/topprismdata/cultivating-ml-agent/kaggle-oof-lb-validation-protocol"><img src="https://agentmods.dev/badge/skills/topprismdata/cultivating-ml-agent/kaggle-oof-lb-validation-protocol/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/topprismdata/cultivating-ml-agent/kaggle-oof-lb-validation-protocol"><img src="https://agentmods.dev/badge/skills/topprismdata/cultivating-ml-agent/kaggle-oof-lb-validation-protocol.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00214 | $0.02780 |
| Opus 5 | $0.00107 | $0.01390 |
| Sonnet 5 | $0.00043 | $0.00556 |
| Haiku 4.5 | $0.00021 | $0.00278 |
Grade A, and why
kaggle-oof-lb-validation-protocol scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 226 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Kaggle OOF/LB Validation Protocol
Problem
OOF (Out-of-Fold) cross-validation score is the standard proxy for LB (Leaderboard) performance. But the gap between OOF and LB can be large, unpredictable, and asymmetric. Submitting based on OOF alone wastes quota when the LB doesn't confirm.
The 5 Sources of OOF/LB Gap
Source 1: Overfitting to CV Fold Structure
Symptom: OOF improves with each tuning round, but LB plateaus or drops.
Root cause: Your CV folds have a specific distribution that your tuning
exploits. The LB test set has a different fold structure.
Magnitude: 0.001-0.005 (small but cumulative)
Fix: Use GroupKFold or StratifiedKFold matching the test set's structure.
For time series, use TimeSeriesSplit with proper temporal ordering.
Evidence (TPS May 2022):
| Round | OOF AUC | LB AUC | Gap |
|---|---|---|---|
| R5 | 0.786 | 0.867 | +0.081 |
| R8 (same code) | 0.867 | 0.880 | +0.013 |
| R13 (stacking) | 0.921 | 0.867 | -0.054 |
R5→R8: same code, 1.37pp gap = pure sampling noise. R13: stacking overfit OOF.
Source 2: Distribution Shift Between Train and Test
Symptom: OOF is stable but LB is consistently lower (or higher). Root cause: Test set comes from a different distribution (time period, geography, population). Magnitude: 0.005-0.05 (can be large) Fix: Run adversarial validation. If AUC > 0.55, there's detectable shift. Consider purifying training data to match test distribution.
Evidence (Store Sales):
- Adversarial AUC = 0.52 → train/test aligned, stop purifying
- lag features caused 10× worse LB because test set had different lag structure
Source 3: Small Sample Size → OOF Inflation
Symptom: OOF on N<1000 rows shows high variance; small changes look significant. Root cause: With few validation samples, each fold's score has high variance. Magnitude: 0.01-0.09 (can dominate real signal) Fix: Require N≥400 games/fold before trusting OOF. Use bootstrap confidence intervals, not point estimates.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 226 lines · 214 tokens per session scan A 88daff65e618
kaggle-oof-lb-validation-protocol is a skill published in the GitHub repository topprismdata/cultivating-ml-agent (5 stars, last pushed 13d ago), licensed MIT. It adds 214 tokens to every session and 2,780 once invoked, about $0.0011 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
huggingface-hub
Hugging Face Hub CLI (hf) — search, download, and upload models and datasets, manage repos, query datasets with SQL, deploy inference endpoints, manage Spaces and buckets.
writing
A writing guide for turning verified facts and calculations into finished text for a specific audience. It follows the requested language, structure, and length.
regex-mastery
Use this skill when writing regular expressions, debugging pattern matching,optimizing regex performance, or implementing text validation. Triggers on regex, regular expressions, pattern matching, lookahead, lookbehind, named groups, capture groups, backreferences, and any task requiring text pattern matching.
ws-ckpt
A Linux-only tool for creating and managing workspace snapshots. A snapshot is a saved view of files that can later be compared, restored, listed, or deleted.
food-order
Reorder previous Foodora orders, preview cart contents, and track delivery ETA/status with ordercli. Use when the user wants to reorder food, check delivery status, or browse recent Foodora order history. Never confirm an order without explicit user approval.
mnemon
Persistent memory for MiniMax Code. Recall durable context, store important facts and decisions, and link related memories with the mnemon CLI.