Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add ralfyishere/rules-with-receipts --skill empirical-validationgit clone --depth 1 https://github.com/ralfyishere/rules-with-receiptsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/ralfyishere/rules-with-receipts/empirical-validation)<a href="https://agentmods.dev/skills/ralfyishere/rules-with-receipts/empirical-validation"><img src="https://agentmods.dev/badge/skills/ralfyishere/rules-with-receipts/empirical-validation/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/ralfyishere/rules-with-receipts/empirical-validation"><img src="https://agentmods.dev/badge/skills/ralfyishere/rules-with-receipts/empirical-validation.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00117 | $0.01711 |
| Opus 5.5 | $0.00047 | $0.00684 |
| Sonnet 5.5 | $0.00023 | $0.00342 |
| Haiku 4.5 | $0.00012 | $0.00171 |
Grade A, and why
Empirical Validation scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 114 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Empirical Validation
Purpose
Claims of efficacy are cheap to make and expensive to trust. An inherited service "has a 74% success rate"; a new rule "sharpens the model"; a vendor "cuts errors 40%." The default failure is to reason about whether these hold — plausibility-check them, argue both sides — and then build on them. Reasoning cannot distinguish a real effect from an overfit artifact; only measurement can. This skill makes the reflex: when a claim is load-bearing, find the cheapest experiment that could falsify it, run it against real data with confidence intervals, and let the result decide — before you invest in it or ship it.
It is the difference between "this metric looks predictive" and "over 52,000 samples it's 49.8%, CI excludes nothing"; between "the new rule should help" and "14-rule flagged 9/12, 15-rule 1/9 — it hurt." Both verdicts were unavailable to argument and decisive to data.
When to use this skill
- About to rely on an inherited/abandoned system's claimed advantage (revival, due diligence).
- About to ship a change to something whose value is empirically established (a proven prompt, snippet, model, config, few-shot set) — verify the change didn't erode it.
- A performance number appears with no committed, re-runnable artifact behind it (docstring win rate, vendor ROI, "we saw a lift").
- A decision worth real money/time/reputation rests on "X works."
- Someone asks "does this actually work?" and the honest answer is "nobody measured."
When NOT to use
- No efficacy claim is load-bearing — you're not betting on whether something works.
- The claim is already backed by a reproducible artifact you can inspect (read it instead).
- Pure correctness questions about your own output — that's
adversarial-verify. - The cost of the experiment exceeds the cost of being wrong (rare; usually the cheap experiment is far cheaper than the misplaced investment — check before assuming this).
The procedure
- Name the one claim the decision lives or dies on, in falsifiable terms. Not "the bot is good" but "signal S predicts the 5-min direction >52% after costs."
- Find the cheapest ground-truth for it. Free public data, a held-out slice, a
historical log, a small controlled run. The best experiments cost cents and minutes
(a free public dataset pull; 12
claude -pcells). Ask: what's the least I can gather that could prove this false? - Design to isolate and to falsify. Change exactly one variable (controlled A/B: same everything, differ only in the thing under test). Prefer a design where a null result is meaningful. Interleave arms so a partial run stays balanced.
- Run it and compute uncertainty. Report rates with confidence intervals, not point estimates; state n; flag thin cells. A "0%" on n=12 is not the same as on n=3,000. Compute the CI on the independent unit (window, user, day), not the raw row count — correlated rows inflate n and manufacture false precision.
- Adversarially check the result before trusting it: fat tails / worst cases (not just the average), sample-regime limits, whether your proxy equals the real settlement variable, whether a "positive" is an upper bound (naive fill/selection bias).
- Act on the verdict, and publish the artifact. The experiment script + numbers are the receipt — reproducible, committed. Let a kill be a kill and a pass be a pass; update the plan, don't re-litigate the data.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 114 lines · 0 tokens per session scan A a6491a5a4118
Empirical Validation is a skill published in the GitHub repository ralfyishere/rules-with-receipts (2 stars, last pushed 3mo ago), licensed MIT. It adds 117 tokens to every session and 1,711 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
alterlab-qiime2-amplicon
Runs 16S/ITS amplicon (microbiome) analysis with the QIIME 2 distribution (2026.7; the "amplicon" distribution was renamed "qiime2" in 2026.4) in the correct order: manifest import, cutadapt trim-paired primer removal BEFORE dada2 denoise-paired (trunc-len chosen from the demux quality .qzv), feature-classifier…
alterlab-imaging-data-commons
Query and download public cancer imaging data from the NCI Imaging Data Commons (IDC) using the idc-index Python package, filtering by metadata, visualizing in-browser, and checking licenses, with no authentication required. Use when obtaining large-scale radiology (CT, MR, PET) or digital pathology DICOM datasets for…
alterlab-cobrapy
Build and analyze genome-scale constraint-based metabolic models with COBRApy — flux balance analysis (FBA), flux variability analysis (FVA), gene and reaction knockouts, flux sampling, and SBML model I/O. Use when simulating metabolic networks, predicting growth or knockout phenotypes, or running systems-biology and…
alterlab-phylogenetics
Build phylogenetic trees end-to-end from raw sequences — MAFFT multiple sequence alignment, optional TrimAl trimming, IQ-TREE 3 maximum-likelihood inference with model selection and bootstraps, FastTree for large datasets, then visualize with ETE3 or FigTree. Use when reconstructing trees from sequences (FASTA) for…
alterlab-datamol
Wraps RDKit in a high-level, pandas-friendly datamol interface with sensible defaults for everyday drug discovery — SMILES/SDF loading into DataFrames, molecule standardization, descriptors, fingerprints, Butina clustering, 3D conformer generation, scaffold analysis, and parallel batch processing, returning native…
alterlab-deepchem
Runs molecular machine learning with DeepChem — diverse featurizers, pre-built MoleculeNet benchmark datasets, and pre-trained models (ChemBERTa, GROVER) for property prediction (ADMET, toxicity, solubility) via traditional ML or graph neural networks. Use when running end-to-end molecular ML experiments that need…