ab-testing

A/B Testing

Workflow

1. Hypothesis Generation

Format: If we [change], then [metric] will [improve/decrease] by [amount], because [rationale].

Example: If we shorten the signup form from 5 fields to 3, then signup completion rate will increase by 15%, because friction reduction at high-intent moments increases conversion.

2. Prioritization

ICE framework (quick):

Factor	Score 1-10	Definition
Impact	1-10	How much will it move the metric?
Confidence	1-10	How sure are we it'll work?
Ease	1-10	How fast/cheap to implement?
ICE Score		(I + C + E) / 3

RICE framework (more rigorous):

Factor	Definition
Reach	How many users affected per quarter?
Impact	Expected effect size (0.25, 0.5, 1, 2, 3)
Confidence	% sure (100%, 80%, 50%)
Effort	Person-weeks to implement
RICE Score	(R × I × C) / E

3. Sample Size Calculation

Formula:

n = (Z_α/2 × √(2p̄(1-p̄)) + Z_β × √(p₁(1-p₁) + p₂(1-p₂)))² / (p₂ - p₁)²

Where:
  p₁ = baseline conversion rate
  p₂ = expected conversion rate (baseline × (1 + MDE))
  p̄  = (p₁ + p₂) / 2
  Z_α/2 = 1.96 (for 95% confidence)
  Z_β   = 0.84 (for 80% power)

Quick reference table:

Baseline rate	MDE (relative)	Sample per variant
2%	10%	78,000
2%	20%	20,000
5%	10%	30,000
5%	20%	7,700
10%	10%	14,300
10%	20%	3,700
20%	10%	6,300
20%	20%	1,600

Test duration:

Days needed = (Sample per variant × 2) / Daily traffic to test page

Minimum: run ≥ 1 full business cycle (usually 7 or 14 days) so every day-of-week and any weekly purchase/payday rhythm is represented; never stop mid-week even if the sample target is hit early.

Run length is driven by power and cycles, not a fixed cap. There is no universal "stop at 4 weeks" rule — B2B, marketplace, pricing, retention, and low-traffic tests routinely need 6–12+ weeks. The real risks in a long test are exposure/sample drift (the population changes — new acquisition channels, seasonality, holidays) and novelty/primacy effects (returning users react to the change for a few weeks, then revert). Mitigate by:

• Decide the fixed horizon up front from the sample-size calc (or use a sequential design, below). Do not let "it's been a month" become the stopping rule.
• Plot the daily cumulative lift; a stable, flattening curve signals novelty has worn off, a still-trending one means keep running.
• For novelty-prone changes (UI redesigns, new features), report new-user vs returning-user segments separately (pre-registered — see segmentation in §6) and consider a long-running holdback to measure the durable effect.
• If you must change the experiment design or population mid-flight, stop and restart as a new test rather than reinterpreting the old one.