experiment-design

Experiment Design

Overview

Experiments are the currency of product discovery. A well-designed experiment tells you whether an assumption is valid — fast and cheap — before you invest in building. This skill teaches the PoL (Proof of Learning) method and experiment design patterns.

When to Use

• You have a hypothesis that needs testing before committing to build
• You want to de-risk a solution before engineering investment
• Stakeholders are debating assumptions — design an experiment to settle it
• You need evidence for a PRD or business case
• Don't use for: well-proven problems (build it), regulatory requirements, or when time-to-market is the only priority

Instructions

1. Define the Hypothesis

Use the format: We believe [solution] will [outcome]. We'll know we're right when [evidence/signal].

Example: "We believe adding a progress bar to the setup wizard will increase completion rate from 30% to 50%. We'll know we're right when we see >45% completion rate over a 2-week A/B test."

Break it down:

• Assumption: What are we betting on?
• Risk level: How confident are we? (🔴 low / 🟡 medium / 🟢 high)
• Evidence threshold: What specific number or signal proves us right?

2. Choose the Cheapest Valid Experiment

From cheapest to most expensive:

1. Customer conversation — "Would you use this?" (cheap but unreliable)
2. Concept test — Show a mockup, ask about behavior change
3. Landing page test — Build a landing page, measure signups
4. Fake door test — A button that leads to "coming soon" — measure clicks
5. Concierge test — Manually deliver the service
6. Wizard of Oz — Pretend the product works, manually operate behind the scenes
7. Prototype test — Clickable prototype with usability tasks
8. A/B test — Real code, real users, real metrics (most expensive)

Pick the cheapest one that gives you a valid signal for your hypothesis.

3. Design the Experiment Card

---
**Hypothesis**: We believe [solution] will [outcome]
**Experiment**: Describe what you'll do
**Success criteria**: What specific signal proves the hypothesis?
**Duration**: How long will you run the experiment?
**Cost**: Time, money, people needed
**Risks**: What could invalidate the results?
**Decision**: If success → build. If failure → pivot or kill.
---