experiment-design

Design experiments that actually prove something. Most A/B tests fail because they test vague ideas, run too short, or peek at results. A well-designed experiment has a clear hypothesis, adequate power, and a pre-committed analysis plan.

Hypothesis Template

Every experiment starts with a written hypothesis before any work begins:

"If we [make this specific change] for [this audience], then [this metric] will [change in this direction] by [this amount], because [this reason based on evidence]."

Example:

"If we replace the 5-step onboarding wizard with a single guided first-project flow for new signups, then 7-day activation rate will increase from 23% to 35%, because 4/6 interviewed users said they wanted to 'just start using it' not 'set everything up first.'"

Every part matters:

• Specific change: Not "improve onboarding" — the exact change
• Audience: Who sees this? New users only? Free tier only?
• Metric + direction + amount: A number you'll measure
• Because: The evidence-based reason. No evidence = no experiment.

Experiment Design

1. Primary Metric

One metric the experiment is designed to move. Not three. One. Additional metrics are guardrails.

2. Guardrail Metrics

Metrics that must NOT degrade. These prevent "winning" by breaking something else.

3. Sample Size

Calculate BEFORE running. Use a sample size calculator with:

• Baseline conversion rate (current number)
• Minimum detectable effect (smallest change worth caring about)
• Statistical significance (95% is standard)
• Power (80% minimum)

If you need 50,000 users and you get 500/week, the experiment will take 100 weeks. Either increase the MDE or don't run the experiment.

4. Duration

Run for at least one full business cycle (usually 1-2 weeks minimum) to capture day-of-week effects. NEVER run less than 7 days.

5. Analysis Plan

Write BEFORE launching: what metric, what threshold, what you'll do if it wins/loses/is inconclusive. Pre-commit to avoid post-hoc storytelling.