measure-experiment-design

Experiment Design

An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal.

When to Use

• Before launching an A/B test to validate a product change
• When testing a hypothesis that requires quantitative validation
• After solution design to validate assumptions before full rollout
• When stakeholders want data-driven evidence for a decision
• To establish a culture of experimentation and learning

When NOT to Use

• The hypothesis itself is not yet articulated -> use define-hypothesis first; this skill designs the test for a claim you already have
• You are analyzing a completed experiment -> use measure-experiment-results
• You need the event tracking that will measure the experiment -> use measure-instrumentation-spec
• You are gathering opinions rather than running a controlled test -> use measure-survey-analysis

Instructions

When asked to design an experiment, follow these steps:

1. Articulate the Hypothesis Write a clear, testable hypothesis in the format: "We believe [change] for [users] will [outcome] as measured by [metric]." One hypothesis per experiment - if you're testing multiple things, run multiple experiments.

2. Define the Variants Describe the control (current experience) and treatment (new experience) in sufficient detail. Include screenshots, mockups, or precise descriptions so anyone can understand what users will see.

3. Choose Primary and Secondary Metrics Select one primary metric that will determine success or failure. Add 2-3 secondary metrics to understand the broader impact. Include guardrail metrics to catch unintended negative effects.