Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add Aperivue/medsci-skills --skill model-validationgit clone --depth 1 https://github.com/Aperivue/medsci-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/aperivue/medsci-skills/model-validation)<a href="https://agentmods.dev/skills/aperivue/medsci-skills/model-validation"><img src="https://agentmods.dev/badge/skills/aperivue/medsci-skills/model-validation.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00149 | $0.02674 |
| Opus 5 | $0.00075 | $0.01337 |
| Sonnet 5 | $0.00030 | $0.00535 |
| Haiku 4.5 | $0.00015 | $0.00267 |
Grade A, and why
model-validation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 173 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Model-Validation Skill
Purpose
This skill pressure-tests the validation study for an engineer-built medical-imaging model — the
common case where a clinical team receives a trained segmentation / classification / detection model
from an engineering collaborator and must validate it and write it up. It is the imaging-model
specialization of /design-study: where design-study covers general validity and
design-ai-benchmarking covers AI-versus-human-expert reader studies, this skill owns the
partition, leakage, reproducibility, and metric-selection mechanics that decide whether a reported
Dice / AUROC / sensitivity is trustworthy.
It is advisory and deterministic-audit only. It writes decision notes and runs a stdlib gate on the split table; it never builds, trains, or alters the model, and it never replaces MONAI / nnU-Net / TorchIO — those produce the model, this validates and publishes it.
When to use
- A trained imaging model (in-house, vendor, or open-weights) needs a clinical-validation study designed or audited before submission.
- You have, or can produce, the split-assignment table (which patient went to train / val / test).
When NOT to use
- Building or training the model → out of scope (integrate MONAI / nnU-Net).
- AI-versus-human-expert reader study →
/design-ai-benchmarking. - LLM / MLLM evaluation →
/mllm-eval(when available). - General study/validity review →
/design-study. - Statistical execution (DeLong, ICC, bootstrap CIs, calibration tables) →
/analyze-stats. - Item-by-item reporting-guideline audit of a finished manuscript →
/check-reporting. - Reviewing a finished manuscript →
/self-reviewor/peer-review(which load the MD0–MD11 reviewer-side probe).
Workflow
The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the
internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed
variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in
${CLAUDE_SKILL_DIR}/references/validation_design.md (load on demand). The patient-disjointness
verdict itself is proven by scripts/check_split_leakage.py (Phase 2), not from that prose.
What ships with it
14 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- references/validation_design.md 13 KB
- scripts/check_split_leakage_challenge/expected/clean.txt 358 B
- scripts/check_split_leakage_challenge/expected/leak.txt 470 B
- scripts/check_split_leakage_challenge/fixture/split_seed.txt 3 B
- scripts/check_split_leakage_challenge/fixture/splits_clean.csv 150 B
- scripts/check_split_leakage_challenge/fixture/splits_leak.csv 145 B
- scripts/check_split_leakage_challenge/problem.md 2.3 KB
- scripts/check_split_leakage_challenge/verify.sh 1.9 KB runs code
- scripts/check_split_leakage.py 11 KB runs code
- skill.yml 3.0 KB
- tests/fixtures/leak_subject.csv 62 B
- tests/fixtures/noseed_clean.csv 50 B
- tests/fixtures/single_partition.csv 55 B
- tests/test_split_leakage.sh 3.9 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 7d ago First seen · 173 lines · 149 tokens per session scan A be75cfcf232c
model-validation is a skill published in the GitHub repository Aperivue/medsci-skills (283 stars, last pushed yesterday), licensed MIT. It adds 149 tokens to every session and 2,674 once invoked, about $0.0007 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
clinical-trials-search
Search ClinicalTrials.gov with natural language queries. Find clinical trials, enrollment, and outcomes using Valyu semantic search.
shidi
A Chinese-language research assistant role that turns a user’s ideas into literature reviews, experiment plans, figures and organised data.
clinical-research
Use when designing a prospective clinical study before submission — selecting and classifying endpoints (primary / key-secondary / exploratory, with surrogate-endpoint flagging), estimating sample size and power for two-arm designs (means / proportions / survival), or scoring a study plan for feasibility and a GO /…
sr-search-record
An automated literature-review search and screening workflow using OpenAlex, an open database and API for scholarly research, and Zotero, a reference manager.
critical-paper-reading
A paper-reading workflow for critically examining one research paper from a PDF or Zotero item. Zotero is a tool for organising research papers and references.
topic-hypothesis-testing
A Chinese-language academic research workflow for turning a research idea into testable hypotheses about cause-and-effect relationships.