bayesian-experiment-reader

Bayesian experiment reader

A frequentist p-value answers a question stakeholders don't ask: "if the variants were identical, how surprising would this data be?" What they actually want is "what's the chance the variant is better?" and "if I ship it and I'm wrong, how bad is it?" Bayesian inference answers both directly. This skill encodes that math and the decision rule it enables.

It pairs with experiment-result-reader. Run that one first for the frequentist read and the setup checks (SRM, mix shift, peeking). Run this one to translate the same per-variant counts into a posterior probability and a ship/hold/kill decision.

When NOT to use this

• The setup isn't clean. SRM, exposure-event gaps, or mix shift contaminate Bayesian math just as badly as frequentist math. Fix the setup first via experiment-result-reader's Phase 1 and Phase 4.
• The conversion metric is heavily right-skewed and you only have a handful of conversions per variant (e.g. revenue per user with three whales). The Normal-Normal model assumes approximately normal sampling distributions; small-sample skew breaks it. Either log-transform, bucket into a proportion, or wait for more data.
• The user wants to design a new experiment. Sample-size planning under a Bayesian framework is a different problem (expected loss under prior + planned n). This skill reads results, it doesn't plan them.
• The user wants a single number to defend a decision in a hostile review. Bayesian outputs are inherently prior-conditional. If the room won't accept "we used a Beta(1,1) prior," stick with the frequentist read.

Why Bayesian beats frequentist for shipping decisions

Three concrete reasons, not aesthetics:

1. Posterior probability is the actual decision variable. "P(variant > control) = 0.97" maps directly to a ship decision. A p-value of 0.03 doesn't: it's the probability of the data under a null hypothesis, which is not what anyone is choosing between.
2. No peeking penalty. Bayesian posteriors update coherently as data arrives. There's no alpha-spending budget to blow, no sequential-testing correction required for the math to be valid. (You still want a pre-committed decision rule, but the math itself doesn't degrade.)
3. Expected loss is the risk side of the ledger. A variant can be 96% likely to be better and still be a bad ship if the 4% downside is catastrophic (a revenue test where the worst-case is −15%). Frequentist methods don't carry that asymmetry; Bayesian expected loss does.