ab-test-setup

Core Principles

1. Start with a Hypothesis

• Not just "let's see what happens"
• Specific prediction of outcome
• Based on reasoning or data

2. Test One Thing

• Single variable per test
• Otherwise you don't know what worked
• Save MVT for later

3. Statistical Rigor

• Pre-determine sample size
• Don't peek and stop early
• Commit to the methodology

4. Measure What Matters

• Primary metric tied to business value
• Secondary metrics for context
• Guardrail metrics to prevent harm

---

Hypothesis Framework

Structure

Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].

Examples

Weak hypothesis: "Changing the button color might increase clicks."

Strong hypothesis: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

Good Hypotheses Include

• Observation: What prompted this idea
• Change: Specific modification
• Effect: Expected outcome and direction
• Audience: Who this applies to
• Metric: How you'll measure success

---

Test Types

A/B Test (Split Test)

• Two versions: Control (A) vs. Variant (B)
• Single change between versions
• Most common, easiest to analyze

A/B/n Test

• Multiple variants (A vs. B vs. C...)
• Requires more traffic
• Good for testing several options

Multivariate Test (MVT)

• Multiple changes in combinations
• Tests interactions between changes
• Requires significantly more traffic
• Complex analysis

Split URL Test

• Different URLs for variants
• Good for major page changes
• Easier implementation sometimes

---

Sample Size Calculation

Inputs Needed

1. Baseline conversion rate: Your current rate
2. Minimum detectable effect (MDE): Smallest change worth detecting
3. Statistical significance level: Usually 95%
4. Statistical power: Usually 80%