Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add uchicago-dsi/ai-sci-skills --skill leakage-and-splitsgit clone --depth 1 https://github.com/uchicago-dsi/ai-sci-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/uchicago-dsi/ai-sci-skills/leakage-and-splits)<a href="https://agentmods.dev/skills/uchicago-dsi/ai-sci-skills/leakage-and-splits"><img src="https://agentmods.dev/badge/skills/uchicago-dsi/ai-sci-skills/leakage-and-splits/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/uchicago-dsi/ai-sci-skills/leakage-and-splits"><img src="https://agentmods.dev/badge/skills/uchicago-dsi/ai-sci-skills/leakage-and-splits.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00064 | $0.00833 |
| Opus 5.5 | $0.00026 | $0.00333 |
| Sonnet 5.5 | $0.00013 | $0.00167 |
| Haiku 4.5 | $0.00006 | $0.00083 |
Grade A, and why
leakage-and-splits scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 73 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Leakage And Splits
The vocabulary below is generic on purpose. A "source group" is whatever repeats: a participant, a site, an animal, a batch, a sensor, a time period, a physical specimen.
Hold Out Whole Groups, Not Rows
- A unit reported as held out must contribute neither gradient nor model/optimizer/checkpoint selection. If an outer objective uses data excluded only from an inner solve, call it adaptation or outer-training data and reserve a separate untouched evaluation split.
- For longitudinal or repeated-measure analyses, assign split eligibility to the whole participant/source group before forming pairs. If any visit is held out, exclude every visit in that group from training. Define sufficiency, permutation, and holdout counts in independent groups rather than in visits or pair combinations.
- Treat raw source identity, not an analysis-row identifier, as the unit. When rows or annotations reuse a source, record that identity, keep splits source-exclusive, report within-source envelopes, and balance cohort summaries over sources.
Do Not Inflate The Denominator
- Do not count overlapping windows, crops, augmentations, repeated rows, or multiscale views of the same physical support as independent evidence. Collapse them to the declared independent unit, or report the dependence explicitly and narrow the claim.
- Permutation controls for group-level conditioning must permute across the grouping unit, not merely across items in batches that may share one source. Record donor/recipient group overlap and require it to be zero for a cross-source claim.
- A leave-one-unit-out agreement gate must compare the omitted unit against an estimate built exclusively from the retained units. Do not compare a retained estimate with a full aggregate that still contains the omitted unit, and include an adversarial two-unit disagreement check before trusting the gate.
Audit Selection-Time Leakage, Not Only Execution-Time Access
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 73 lines · 64 tokens per session scan A 518eb2699511
leakage-and-splits is a skill published in the GitHub repository uchicago-dsi/ai-sci-skills (20 stars, last pushed 4d ago), licensed MIT. It adds 64 tokens to every session and 833 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-22.
Other skills, from other repositories
cli-eval
Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.
model-merging
Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task…
darwinian-evolver
Evolve prompts/regex/SQL/code with Imbue's evolution loop.
dummy-dataset
Generate realistic dummy datasets for testing with customizable columns, constraints, and output formats (CSV, JSON, SQL, Python script). Use when creating test data, building mock datasets, or generating sample data for development and demos.
validate
Validate Semantica pipelines, extraction quality, graph schemas, and ontology consistency. Returns structured error/warning checklists. Uses PipelineValidator, PipelineBuilder.validatepipeline(), GraphValidator, and OntologyValidator. Sub-commands: pipeline, step, dependencies, extraction, graph, ontology, performance.
agent-evals
Build automated evaluation suites for AI agents using golden datasets, rubrics, and regression gates. Use when shipping agent features, validating prompt changes, or gating deployments on quality.