Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add aiopshwang/data-analysis-ml-agent-skills --skill validating-models-and-claimsgit clone --depth 1 https://github.com/aiopshwang/data-analysis-ml-agent-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/aiopshwang/data-analysis-ml-agent-skills/validating-models-and-claims)<a href="https://agentmods.dev/skills/aiopshwang/data-analysis-ml-agent-skills/validating-models-and-claims"><img src="https://agentmods.dev/badge/skills/aiopshwang/data-analysis-ml-agent-skills/validating-models-and-claims/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/aiopshwang/data-analysis-ml-agent-skills/validating-models-and-claims"><img src="https://agentmods.dev/badge/skills/aiopshwang/data-analysis-ml-agent-skills/validating-models-and-claims.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00058 | $0.00837 |
| Opus 5 | $0.00029 | $0.00418 |
| Sonnet 5 | $0.00012 | $0.00167 |
| Haiku 4.5 | $0.00006 | $0.00084 |
Grade A, and why
validating-models-and-claims scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 11d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 72 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Validating Models and Claims
Decide what the available evidence actually supports. Treat a metric, chart, or successful run as evidence for a bounded claim, not as proof that the full system works.
Start with the claim
Write each material claim in a falsifiable form before selecting evidence:
- the decision or action the claim will influence;
- the population, unit of analysis, prediction horizon, and operating conditions;
- the comparator and practical improvement threshold;
- the cost of false positives, false negatives, abstention, and delay;
- the evidence that would refute or narrow the claim.
If these are unresolved and materially change the evaluation, surface the choice instead of silently selecting a convenient definition.
Build an evidence contract
For every claim, map the claim's scope to evidence of equal scope. A component test cannot establish end-to-end readiness; aggregate accuracy cannot establish subgroup safety; retrospective fit cannot establish future performance.
Separate at least these evidence sources when they exist:
- development evidence used to choose features, thresholds, or models;
- untouched or independently collected validation evidence;
- human-reviewed ground truth and its adjudication record;
- operational evidence produced under the real inference path.
Do not describe reused tuning data as independent validation. Record every contact with the evaluation set, including manual error review that influenced a subsequent model choice.
When label correctness depends on expertise or judgment, preserve the human ground-truth process: reviewer qualifications, instructions, blinding, disagreement rate, adjudication, unresolved cases, and sampling frame. Model agreement with noisy labels is not accuracy against reality.
Validate in layers
Use the smallest set of checks that covers the actual claim:
- Semantic validity: target, timestamps, joins, exclusions, and units mean what the claim assumes.
- Evaluation integrity: split boundaries, leakage controls, comparator, threshold selection, and metric implementation are valid.
- Performance: report uncertainty and practical effect size, not only a point estimate.
- Slices and errors: inspect decision-relevant subgroups, rare cases, temporal drift, missingness patterns, and high-cost errors.
- Operational fidelity: when the claim includes deployment or operational behavior, exercise the same preprocessing, feature availability, threshold, and output interpretation used in practice. Otherwise record why this layer is not applicable.
- Claim calibration: narrow the final language to what all completed evidence supports.
What ships with it
3 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 11d ago First seen · 72 lines · 58 tokens per session scan A 12548bcea02f
validating-models-and-claims is a skill published in the GitHub repository aiopshwang/data-analysis-ml-agent-skills (12 stars, last pushed 14d ago), licensed MIT. It adds 58 tokens to every session and 837 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
eu-ai-act-readiness
Build a preliminary, evidence-based EU AI Act readiness assessment across AI-system inventory, territorial scope, operator roles, prohibited-practice screening, risk classification, transparency, high-risk controls, general-purpose AI obligations, governance, and implementation milestones. Use when an organization…
data-labeling
Set up and manage data labeling workflows using manual annotation tools, semi-automated pipelines, active learning, and programmatic weak supervision. Use when the user requests data labeling or provides relevant inputs for this workflow.
hyperparameter-tuning
Optimize machine learning model hyperparameters using grid search, random search, Bayesian optimization, and Hyperband to maximize model performance within a compute budget. Use when the user requests hyperparameter tuning or provides relevant inputs for this workflow.
model-training
Train machine learning models end-to-end, covering data loading, preprocessing, architecture selection, training loops, validation, and checkpointing. Use when the user requests model training or provides relevant inputs for this workflow.
context-injection
Place trusted contextual information into prompts or agent state using explicit boundaries, provenance, and templates. Use when relevant context has already been selected and must be inserted safely; use context-retrieval to find it or context-optimization to choose and order it.
context-retrieval
Retrieve relevant information from a knowledge base using semantic, keyword, or hybrid search to ground a query. Use when the task starts with a corpus or index that must be searched; use context-ranking when candidate chunks already exist and only need ordering.