experiment

Experiment Loop Skill (XLOOP)

Skill for running automated iterative improvement loops that optimize a measurable metric while keeping changes simple and reversible.

Overview

The experiment loop (auto experiment) runs an agent-driven cycle:

1. Measure baseline metric
2. Ask an executor agent to make one focused change
3. Measure the new metric
4. Decide: keep (commit) or discard (reset)
5. Check circuit breaker and simplicity gate
6. Repeat until MaxIterations or circuit break

Setup

# Initialize experiment branch (requires clean worktree)
auto experiment init --session-id my-session

# Verify metric command works
auto experiment metric \
  --metric 'go test -bench=. ./... | grep ns/op | awk "{print \"{\\\"metric\\\":\"$3\"}\"}"' \
  --metric-key metric

Full Loop Configuration

auto experiment init --session-id opt-$(date +%s)

# Then invoke the loop via the agent skill below

Agent Orchestration Pattern

The experiment loop is driven by invoking an executor agent repeatedly. Pass the full history context on each call so the agent can learn from past iterations.

## Experiment Loop — Iteration {N}

### Config
- MetricCmd: {cmd}
- Direction: {minimize|maximize}
- Target: {files}
- Scope: {files or "same as target"}
- SimplicityThreshold: {threshold}

### History (last 5 results)
{JSON array of recent Result objects from `auto experiment record`}

### Baseline: {baseline_value} {unit}
### Best so far: {best_value} at iteration {best_iter}

### Your task
Make ONE focused change to the target files that should improve the metric.
Do NOT modify files outside the allowed scope.
After making your change, run:
  auto experiment commit --iteration {N} --description "your change description"
Then output the description in the last line.

Keep / Discard Decision

After each executor run, measure the metric and decide:

new_value = RunMetricMedian(cfg, cmd)
simplicity = CalculateSimplicity(baseline, new_value, linesAdded, linesRemoved, direction)

if direction.IsBetter(new_value, best_value):
    if simplicity >= cfg.SimplicityThreshold:
        status = "keep"
        best_value = new_value
        circuit_breaker.Record(true)
    else:
        status = "discard"   # improvement too small relative to code complexity
        ResetToCommit(last_keep_hash)
        circuit_breaker.Record(false)
else:
    status = "discard"
    ResetToCommit(last_keep_hash)
    circuit_breaker.Record(false)