managing-experiment-lifecycle

Managing experiment lifecycle

This skill covers experiment state transitions — what each action does, when to use it, and how it affects variant assignment and analysis.

State diagram

draft ──launch──▶ running ──end──▶ stopped ──archive──▶ archived
                  │ │   ▲              │
                  │ pause resume  ship_variant
                  │ │   │         (also ends if running)
                  │ ▼   │
                  │ paused (flag inactive, still "running" status)
                  │
                  ├─freeze_exposure──▶ exposure_frozen ──unfreeze_exposure──▶ running
                  │                    (enrollment closed, metrics keep flowing)

Any non-draft state ──reset──▶ draft

Actions and their implications

For each action, the two key questions:

1. Who sees what variant? (user perspective)
2. Who is in my analysis? (statistical perspective)

Launch (experiment-launch)

Transitions draft → running. Activates the feature flag and sets start_date.

• Preconditions: must be in draft, flag needs 2-20 multivariate variants (no specific key required; the baseline defaults to "control" when present, else the first variant)
• Pre-launch checklist: has at least one metric? Variants correct? Flag implemented in code?
• Variants: users start being bucketed into variants based on the configured split
• Analysis: data collection begins from start_date

No request body needed.

One optional item worth a single mention at launch, when the change is user-facing and substantial: a short survey, shown when users finish the experimented flow (e.g. triggered by the form's submit event), collects qualitative feedback (a rating, an optional comment) alongside the metrics, from day one. Offer it once as setup advice, drop it if declined, and never let it delay the launch. Do not raise it at end or ship-variant time — there it reads as a gate on rolling out. → See references/qualitative-feedback.md in [[diagnosing-experiment-results]]