ab-test-design

A/B Test Design

The Law

A TEST WITHOUT A STOPPING RULE RUNS UNTIL THE RESULT LOOKS GOOD.
"We'll stop when we see something significant" is peeking — it inflates false positive rates until the next meaningless spike ships to production.
Hypothesis + metric + sample size + duration + stopping rule + decision rule IS an experiment.

When to Use

Trigger before:

• Running any A/B or multivariate test that will produce a ship/no-ship decision
• Adding feature flag logic that will split users into groups
• Launching any "let's try this and see what happens" experiment

When NOT to Use

• Observational studies with no control group (not an A/B test — use a different analysis method)
• Rollouts to < 1% of users as a smoke test (no statistical inference intended — label it as such)
• Shadow mode tests where both groups see the same experience (monitoring, not experimentation)

The Six Required Elements

1 — Hypothesis

One testable claim in the form: If [change], then [metric] will [increase/decrease] by [magnitude] because [mechanism].

If we [specific change to control],
then [primary metric] will [direction] by [minimum detectable effect],
because [causal mechanism].

What does NOT count:

• "We think this will improve conversion" — no direction, no magnitude, no mechanism
• "Let's test it and see" — not a hypothesis; a hypothesis is a prediction, not an openness to observation

The mechanism matters. A hypothesis without a mechanism cannot be learned from even if it's correct — you won't know why.

2 — Primary Metric

The single metric that defines success or failure for this test.

• One primary metric only — multiple primary metrics require Bonferroni correction and usually indicate an unclear hypothesis
• Must be measurable at the session or user level (not aggregated)
• Must have a baseline from at least the last 14 days of data

Secondary metrics (guardrails and diagnostics — max 5): watched but do not determine the decision.