experiment

/experiment — Metric-Driven Optimization Loop

Inputs

The user provides three things:

1. scope: Files to modify (glob pattern, e.g., "src/api/**/*.ts")
2. metric: Shell command that outputs a single number (e.g., npm run build 2>&1 | tail -1 | grep -oP '\d+')
3. budget: Iteration cap (default: 5) or time cap (e.g., "10 minutes")

If any input is missing, ask for it. The metric MUST output a single number to stdout.

Protocol

Step 1: BASELINE

1. Stash any uncommitted changes (restore on exit)
2. Run the metric command. Record the baseline value.
3. Determine direction: does lower = better (bundle size, error count) or higher = better (FPS, test count)? Ask the user if ambiguous.
4. Log: Baseline: {value} ({metric command})

Step 2: ITERATE

For each iteration (up to budget):

1. Create isolation: Spawn a sub-agent in a worktree (isolation: "worktree")
2. Propose change: The agent modifies files within scope to improve the metric. Provide context: baseline value, metric direction, scope, what previous iterations tried.
3. Measure: Run the metric command in the worktree (via node scripts/run-with-timeout.js 300)
4. Gate: Run typecheck (also via timeout wrapper). If it fails, discard immediately.
5. Evaluate:
   • Improved? → KEEP. Merge the worktree branch. New baseline = new value.
   • Same or worse? → DISCARD. Delete the worktree.
6. Log iteration:

   Iteration {N}: {value} ({delta from baseline}) → {KEEP|DISCARD}
   Change: {one-line description of what was tried}
   

Step 3: CONVERGENCE CHECK

After each iteration, check:

• Local optimum: Last 3 iterations all discarded → stop ("no more improvements found")
• Diminishing returns: Last kept improvement was < 0.5% → stop ("diminishing returns")
• Budget exhausted: Iteration count or time exceeded → stop

Step 4: REPORT

Write results to .planning/research/experiment-{slug}.md:

# Experiment: {Description}

> Metric: `{command}`
> Direction: {lower|higher} is better
> Scope: {glob pattern}
> Budget: {N iterations}
> Date: {ISO date}

## Results

| Iteration | Value | Delta | Verdict | Change |
|-----------|-------|-------|---------|--------|
| baseline  | {N}   | —     | —       | —      |
| 1         | {N}   | {+/-} | KEEP    | {desc} |
| 2         | {N}   | {+/-} | DISCARD | {desc} |

## Outcome
- **Start**: {baseline}
- **End**: {final value}
- **Improvement**: {percentage}
- **Iterations**: {kept}/{total}
- **Stop reason**: {convergence|diminishing|budget}

## Kept Changes
{List of changes that were kept, with commit hashes}