experiment-new

/experiment new

Start a new A/B test experiment.

Instructions

You are helping the user design and run a structured experiment using the ALIVE loop adapted for A/B testing: Design → Validate → Analyze → Decide → Learn.

Follow these steps in order. Do NOT auto-fill any content — ask the user and build the experiment together through conversation.

Step 1: Ask initial questions

Q1: What do you want to test? Understand the change, feature, or hypothesis the user wants to experiment with.

• "What change are you testing? Describe the control (current) and treatment (new)."

Q2: Full or Quick?

• Full Experiment — For experiments with business impact. 5 files, full statistical rigor, checklists.
• Quick Experiment — For low-risk experiments (feature flag toggles, small UI tweaks). Single file, abbreviated.

Guide:

• "Will this experiment influence a product/business decision?" → Full
• "Is this a quick validation or low-stakes test?" → Quick

Q3: What's the primary metric? The single metric that determines success or failure.

• "If you could only look at ONE number to decide whether this worked, what would it be?"
• Reference config.md metrics if available.

Q4: Guardrail metrics? Metrics that must NOT get worse. Reference config.md guardrails.

• "What must NOT break if this experiment succeeds? (e.g., crash rate, page load time, revenue)"

Q5: How long can this experiment run?

• "What's the maximum duration you'd accept? (1 week, 2 weeks, 4 weeks)"
• This informs sample size feasibility.

Step 2: Generate ID

Read .analysis/status.md to determine the next sequence number for today.

• Full Experiment: E-{YYYY}-{MMDD}-{seq} (e.g., E-2026-0215-001)
• Quick Experiment: QE-{YYYY}-{MMDD}-{seq} (e.g., QE-2026-0215-001)

Sequence resets daily, starts at 001.

Step 3: Create files

3A. Full Experiment

Create folder: ab-tests/active/{ID}_{title-slug}/

Generate 01_design.md:

# DESIGN: {title}
> ID: {ID} | Type: 🧪 Experiment | Stage: 📐 DESIGN | Started: {YYYY-MM-DD}

## Hypothesis
- **If** we {change/treatment description},
- **Then** {primary metric} will {direction} by {expected magnitude},
- **Because** {mechanism/reasoning}.

## Experiment Setup
- **Control**: {current experience}
- **Treatment**: {new experience}
- **Variants**: {number of variants, including control}
- **Randomization unit**: user / session / device / other
- **Traffic allocation**: {control %}% / {treatment %}%
- **Target population**: {all users / specific segment}
- **Exclusions**: {who should NOT be in the experiment}

## Metric Structure

### Primary Metric (decision criterion)
| Metric | Current Baseline | MDE (Minimum Detectable Effect) | Direction |
|--------|-----------------|--------------------------------|-----------|
| {metric} | {value} | {absolute or relative change} | ↑ / ↓ |

### Secondary Metrics (additional insight)
| Metric | Baseline | Expected Direction | Why Track |
|--------|----------|--------------------|-----------|
| | | | |
| | | | |

### Guardrail Metrics (must not degrade)
| Metric | Current Value | Acceptable Range | Source |
|--------|--------------|-------------------|--------|
| | | | config.md / custom |

## Sample Size Calculation
- **Test type**: two-sided / one-sided
- **Significance level (α)**: 0.05
- **Power (1-β)**: 0.80
- **Baseline rate/mean**: {from primary metric}
- **MDE**: {from primary metric}
- **Required sample size per variant**: {AI calculates}
- **With current daily traffic ({N}/day)**: ~{X} days needed
- **Feasibility**: ✅ fits within timeline / ⚠️ tight / 🔴 not feasible

> 💡 AI will guide through the calculation:
> - For proportions: n = (Z_α/2 + Z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁-p₂)²
> - For means: n = (Z_α/2 + Z_β)² × 2σ² / δ²
> - Practical shortcut: "Can we get {n} users per variant within {max duration}?"

## Duration & Schedule
- **Start date**: {planned}
- **Minimum duration**: {from sample size calculation}
- **Maximum duration**: {from Q5}
- **Include full week cycles**: Yes (avoid day-of-week effects)
- **Ramp-up plan**: {start at X% → full allocation after Y days} / none

## Risk Assessment
- **Worst-case scenario**: If treatment is bad, what's the impact?
- **Rollback trigger**: {specific condition for emergency stop}
- **Blast radius**: {number of affected users, revenue at risk}
- **Reversibility**: Fully reversible / Partially / Irreversible

## Stakeholders
- **Experiment owner**: {name}
- **Decision maker**: {name/role}
- **Engineering contact**: {name}

## Provenance
- Triggered by analysis: {analysis ID, if any — e.g., "F-2026-0210-001 found that X correlates with Y"}
- Key finding that motivated this experiment:

## Pre-registration
> This section locks the analysis plan BEFORE seeing results (prevents p-hacking).
- Primary metric: {locked}
- Success threshold: {locked — what constitutes a "win"?}
- Analysis method: {locked — frequentist / bayesian / both}
- Segment analysis planned: {list segments you'll check, if any}
- Decision framework: {when to Launch / Kill / Extend / Iterate}

---
{Insert DESIGN checklist from ab-tests/checklists/design.md}