audit-my-experiment

audit-my-experiment

The colleague who runs the checks before you ship the result: computes the validity tests you'd otherwise eyeball, tells you what's broken and what can't be verified yet, and never blesses a number it didn't check.

When to use

Fire when an experiment / A-B / causal result is heading to a decision — even under a consumption ask ("write up our win", "should we ship"). Switch into audit-mode and validate before packaging. Do NOT fire to write up an already-validated result (brief-my-findings), rehearse defending it (defend-my-number), review ONE code object (review-my-query), diagnose why ONE number moved (triage-my-number), audit a whole KB (kb-reconcile), or define a metric (kpi-contract).

The trap this exists to beat

A capable model reads an experiment result and writes the win — and it does the analytical part well: it recognizes Simpson's paradox if segments are shown, catches a narrated novelty story, flags a named peeking admission. Then it does the wrong thing with the checks it should compute. Its instinct is to eyeball the split ("looks roughly 50/50"), glance at p=0.03 and call it significant, and note that "nothing else hit significance" as reassurance. It writes the win under a consumption ask and ships it. The discipline it skips: switch OUT of answer-mode into audit-mode and COMPUTE the checks rather than eyeballing them.

Proven: under the consumption framing ("write up our win"), a cold model shipped a 0.56% SRM — a χ²≈7.8, p≈0.005 — dismissed as "expected noise at scale." The check was never run. The same model waved a peeked p<0.05 through without applying a sequential threshold. Both failures are invisible to a reader; only computation catches them.

The loop

1. Switch to audit-mode + set the target. Recognize an experiment/A-B/causal result headed for a decision, even under a consumption ask. Pin the claim & decision riding on it, the identification strategy (RCT / DiD / other non-RCT — geo, pre/post, IV, synthetic-control), primary metric, arm counts, the stopping story, the metric family, and the minimum-meaningful-effect (MME) — the smallest effect that would change the decision (cost/benefit breakeven or launch bar); elicit it, or mark materiality-unverified (never invent it).
2. Inventory in-hand vs needs-data. Separate checks computable from the summary numbers given (SRM, two-proportion z/CI, multiplicity, power/MDE) from checks needing data not on hand (per-day assignment logs, pre-registration, missing segment cuts).
3. Run the computable checks with the kit — don't eyeball. Execute references/experiment_checks.py with the provided numbers; report each computed statistic. SRM chi-square runs on ANY split.
4. Run the full validity taxonomy (the engine). references/validity-taxonomy.md: design / inference / interpretation layers. Comprehensive thinking, lean output — record what bites.
5. Write the check for anything unverifiable. Exact query/script; mark unverified — needs paste-back. On a pasted run, reconcile (the run wins). Never bless what you can't compute.
6. Grade + gate. Blocking / Latent / Advisory, each with computed evidence + fix direction. A Blocking validity defect gates the ship/brief decision. Materiality rides as its own verdict line — material / immaterial / straddles-MME / materiality-unverified (run classify_materiality) — carried into the handoff; it does NOT gate (a valid experiment can be immaterial), but a ship-ready result is never written up as a material win without it.
7. Emit + route. Write experiment-audit.md; if ship-ready, hand off to brief-my-findings / defend-my-number. KB composition per references/experiment-audit.md. Then stop.