experiment-designer

Experiment Designer

Help product managers design lightweight validation experiments to test hypotheses before committing to full development.

Core Philosophy

Don't build to learn. Experiment to learn, then build.

Experiments answer: "Should we build this?" before investing weeks/months.

---

When to Use This Skill

Trigger experiment design when:

• Confidence < 70% on an initiative
• User asks "should we build X?"
• Feature is expensive (>4 weeks effort)
• Stakeholder request without user validation
• New product/market with uncertainty

Don't experiment when:

• Obvious must-have (e.g., fix critical bug)
• Feature already validated
• Regulatory requirement
• Confidence > 80% with solid data

---

The Experiment Design Workflow

Step 1: Capture the Hypothesis

Ask: "What do you believe to be true that you want to test?"

Help them form a clear hypothesis in IF-THEN format:

Template:

IF [we do X action],
THEN [Y measurable result will happen]
BECAUSE [assumption about users]

Examples:

✅ Good hypothesis: "IF we show dashboard prototype to 10 CFOs, THEN 8/10 will say they'd use it weekly BECAUSE CFOs waste 30min/week on manual reports"

❌ Bad hypothesis (too vague): "IF we build dashboard, THEN users will like it"

Push for specificity:

• What EXACTLY are we testing?
• What EXACT result would confirm it?
• What's the underlying assumption?

Store: hypothesis (IF-THEN-BECAUSE statement)

---

Step 2: Assess Current Confidence

Ask: "How confident are you (0-100%) that this will work?"

This determines which experiment method to use:

Confidence	What it means	Experiment needed
0-30%	Pure speculation	Heavy validation (interviews, prototype)
30-50%	Weak signal	Moderate validation (wizard of oz, concierge)
50-70%	Some evidence	Light validation (feature flags, beta)
70-100%	Strong evidence	Minimal/no experiment, just build