experiment-next

/experiment next

Advance the current experiment to the next stage.

Instructions

Step 1: Identify current experiment

Read .analysis/status.md to find active experiments (Type: 🧪 Experiment).

• If only 1 active Full experiment → select it automatically
• If multiple active → ask user which experiment to advance (show ID + title)
• Quick experiments don't use /experiment next (all sections are in one file)
  • If user selects a Quick, remind them to fill sections in order within the file

Step 2: Determine current stage

Read the experiment folder to see which stage files exist:

• Only 01_design.md → current stage is DESIGN
• Up to 02_validate.md → current stage is VALIDATE
• Up to 03_analyze.md → current stage is ANALYZE
• Up to 04_decide.md → current stage is DECIDE
• All 5 files exist → current stage is LEARN (experiment is complete)

Step 3: Review current stage checklist

Read the current stage's checklist (embedded at the bottom of the current stage file).

Check if there are any 🔴 (stop) items:

• If 🔴 items exist → warn the user: "There are stop signals in your {stage} checklist. Review before proceeding."
• If all items are unchecked → remind: "Don't forget to review the checklist before moving on."
• If items are checked → proceed

Special gate: DESIGN → VALIDATE Before advancing from DESIGN, verify:

• Pre-registration section is filled (this MUST be locked before seeing any data)
• If empty, warn: "Your analysis plan should be pre-registered before starting the experiment. Fill the Pre-registration section in 01_design.md first."

Step 4: Generate next stage file

Based on current stage, generate the next file.

DESIGN → VALIDATE (generate 02_validate.md):

# VALIDATE: {title}
> ID: {ID} | Type: 🧪 Experiment | Stage: ✅ VALIDATE | Updated: {YYYY-MM-DD}

## Pre-Experiment Checks

### AA Test / Historical Balance
- Method: (AA test on pre-period data / historical segment comparison)
- Result: Groups are balanced / not balanced
- Key metrics checked:
  | Metric | Control | Treatment | Δ | Balanced? |
  |--------|---------|-----------|---|-----------|
  | | | | | |

### Sample Ratio Mismatch (SRM)
- Expected ratio: {from 01_design.md}
- Actual ratio: {observed}
- Chi-square test p-value: (p < 0.001 indicates SRM)
- SRM detected? No / Yes → ⚠️ investigate before proceeding

> 💡 SRM indicates a problem with randomization. Common causes:
> - Bot filtering differences between variants
> - Redirect timing differences
> - Initialization bias (treatment takes longer to load)
> If SRM is detected, DO NOT proceed to Analyze. Fix the root cause first.

### Segment Balance
| Segment | Control % | Treatment % | Balanced? |
|---------|-----------|-------------|-----------|
| New users | | | |
| Returning users | | | |
| Platform (iOS/Android/Web) | | | |
| {custom segment} | | | |

### Instrumentation Check
- [ ] Primary metric events logging correctly
- [ ] Secondary metric events logging correctly
- [ ] Guardrail metric events logging correctly
- [ ] No duplicate counting
- [ ] Attribution window correctly set

## External Factors
- Holidays/events during experiment period:
- Planned releases/deployments:
- Marketing campaigns:
- Other active experiments on same population:

## Ramp-Up Log
| Date | Traffic % | Observation | Action |
|------|-----------|-------------|--------|
| {start} | {initial}% | | |

## Validation Summary
- Ready to proceed? ✅ Yes / 🔴 No — {reason}
- Issues found:
- Mitigation:

---
{Insert VALIDATE checklist from ab-tests/checklists/validate.md}