Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add ralfyishere/rules-with-receipts --skill empirical-validationgit clone --depth 1 https://github.com/ralfyishere/rules-with-receiptsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ralfyishere/rules-with-receipts/empirical-validation)<a href="https://agentmods.dev/skills/ralfyishere/rules-with-receipts/empirical-validation"><img src="https://agentmods.dev/badge/skills/ralfyishere/rules-with-receipts/empirical-validation/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/ralfyishere/rules-with-receipts/empirical-validation"><img src="https://agentmods.dev/badge/skills/ralfyishere/rules-with-receipts/empirical-validation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00117 | $0.01711 |
| Opus 5 | $0.00059 | $0.00856 |
| Sonnet 5 | $0.00023 | $0.00342 |
| Haiku 4.5 | $0.00012 | $0.00171 |
Grade A, and why
Empirical Validation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 114 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Empirical Validation
Purpose
Claims of efficacy are cheap to make and expensive to trust. An inherited service "has a 74% success rate"; a new rule "sharpens the model"; a vendor "cuts errors 40%." The default failure is to reason about whether these hold — plausibility-check them, argue both sides — and then build on them. Reasoning cannot distinguish a real effect from an overfit artifact; only measurement can. This skill makes the reflex: when a claim is load-bearing, find the cheapest experiment that could falsify it, run it against real data with confidence intervals, and let the result decide — before you invest in it or ship it.
It is the difference between "this metric looks predictive" and "over 52,000 samples it's 49.8%, CI excludes nothing"; between "the new rule should help" and "14-rule flagged 9/12, 15-rule 1/9 — it hurt." Both verdicts were unavailable to argument and decisive to data.
When to use this skill
- About to rely on an inherited/abandoned system's claimed advantage (revival, due diligence).
- About to ship a change to something whose value is empirically established (a proven prompt, snippet, model, config, few-shot set) — verify the change didn't erode it.
- A performance number appears with no committed, re-runnable artifact behind it (docstring win rate, vendor ROI, "we saw a lift").
- A decision worth real money/time/reputation rests on "X works."
- Someone asks "does this actually work?" and the honest answer is "nobody measured."
When NOT to use
- No efficacy claim is load-bearing — you're not betting on whether something works.
- The claim is already backed by a reproducible artifact you can inspect (read it instead).
- Pure correctness questions about your own output — that's
adversarial-verify. - The cost of the experiment exceeds the cost of being wrong (rare; usually the cheap experiment is far cheaper than the misplaced investment — check before assuming this).
The procedure
- Name the one claim the decision lives or dies on, in falsifiable terms. Not "the bot is good" but "signal S predicts the 5-min direction >52% after costs."
- Find the cheapest ground-truth for it. Free public data, a held-out slice, a
historical log, a small controlled run. The best experiments cost cents and minutes
(a free public dataset pull; 12
claude -pcells). Ask: what's the least I can gather that could prove this false? - Design to isolate and to falsify. Change exactly one variable (controlled A/B: same everything, differ only in the thing under test). Prefer a design where a null result is meaningful. Interleave arms so a partial run stays balanced.
- Run it and compute uncertainty. Report rates with confidence intervals, not point estimates; state n; flag thin cells. A "0%" on n=12 is not the same as on n=3,000. Compute the CI on the independent unit (window, user, day), not the raw row count — correlated rows inflate n and manufacture false precision.
- Adversarially check the result before trusting it: fat tails / worst cases (not just the average), sample-regime limits, whether your proxy equals the real settlement variable, whether a "positive" is an upper bound (naive fill/selection bias).
- Act on the verdict, and publish the artifact. The experiment script + numbers are the receipt — reproducible, committed. Let a kill be a kill and a pass be a pass; update the plan, don't re-litigate the data.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 114 lines · 0 tokens per session scan A a6491a5a4118
Empirical Validation is a skill published in the GitHub repository ralfyishere/rules-with-receipts (2 stars, last pushed 2mo ago), licensed MIT. It adds 117 tokens to every session and 1,711 once invoked, about $0.0006 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
alterlab-pyhealth
Develops, tests, and deploys clinical machine learning models with the PyHealth healthcare AI toolkit. Use when working with electronic health records (EHR), clinical prediction tasks (mortality, readmission, drug recommendation), medical coding systems (ICD, NDC, ATC), physiological signals (EEG, ECG), healthcare…
alterlab-imaging-data-commons
Query and download public cancer imaging data from the NCI Imaging Data Commons (IDC) using the idc-index Python package, filtering by metadata, visualizing in-browser, and checking licenses, with no authentication required. Use when obtaining large-scale radiology (CT, MR, PET) or digital pathology DICOM datasets for…
alterlab-phylogenetics
Build phylogenetic trees end-to-end from raw sequences — MAFFT multiple sequence alignment, optional TrimAl trimming, IQ-TREE 2 maximum-likelihood inference with model selection and bootstraps, FastTree for large datasets, then visualize with ETE3 or FigTree. Use when reconstructing trees from sequences (FASTA) for…
alterlab-deepchem
Runs molecular machine learning with DeepChem — diverse featurizers, pre-built MoleculeNet benchmark datasets, and pre-trained models (ChemBERTa, GROVER) for property prediction (ADMET, toxicity, solubility) via traditional ML or graph neural networks. Use when running end-to-end molecular ML experiments that need…
alterlab-molecular-dynamics
Runs and analyzes molecular dynamics simulations with OpenMM and MDAnalysis — setting up protein and small-molecule systems, assigning force fields, running energy minimization and production MD, and analyzing trajectories (RMSD, RMSF, contact maps, free energy surfaces). Use when simulating protein or ligand…
alterlab-clinicaltrials
Query ClinicalTrials.gov via its API v2 to search trials by condition, drug, location, recruitment status, or phase and retrieve trial details by NCT ID. Use when finding interventional or observational studies, checking trial status and eligibility for patient matching, or exporting clinical trial records for…