experiment

/experiment — Metric-Driven Optimization Loop

Identity

Coordinate bounded metric-driven experiments in isolated checkouts, retaining candidates locally until the declared verification and independent review pass.

Orientation

When to Use

• When you want to optimize a measurable metric (bundle size, error count, test coverage, FPS)
• When you have a clear hypothesis but aren't sure which of several approaches wins
• When manual A/B testing would be too slow or error-prone
• NOT when the goal is subjective ("make it feel better") — the metric must be a number

Inputs

The user provides three things:

1. scope: Files to modify (glob pattern, e.g., "src/api/**/*.ts")
2. metric: Project command that outputs a single number (e.g., node tools/bundle-bytes.cjs)
3. budget: Iteration cap (default: 5) or time cap (e.g., "10 minutes")

Ask only for missing inputs that cannot be resolved from the request or project. The metric MUST exit successfully and output a single finite number to stdout.

Protocol

Step 1: BASELINE

1. Read project instructions and inspect the checkout, index, untracked files, and active worktrees. Record the source commit and dirty-file hashes/status. Leave user work in place: never stash, reset, clean, switch branches, or commit unrelated changes to make an experiment possible.
2. Create a dedicated worktree at an explicit commit, or a separate local clone at that commit. Verify its path, HEAD, and clean status before running commands. If the requested baseline includes authorized working changes, copy only those changes into the isolated checkout, preserving deletions and file contents; verify their hashes and commit the snapshot there. Leave the original index and files untouched. If the intended snapshot is unclear, clarify before copying.
3. Before iterations, write the evaluation plan in the experiment report: source and snapshot commits, allowed source scope, metric command and direction, timeout, repeat/noise policy, and required typecheck, regression and behavioral checks. Record any inapplicable check and its reason now, not after a failure. Ask about direction only if ambiguous. A scalar is a proxy, not acceptance.
4. Freeze the evaluator: record hashes for the metric implementation, test assertions, fixtures/expected outputs, and relevant command/config files. These are outside candidate edit scope, even if a broad glob includes them. Keep the original incumbent commit, evaluator and plan fixed for the run.
5. Pin one file-byte representation before measuring, including checkout filters and line endings. Compare baseline and evaluator hashes with their committed blobs; for commit-based evaluation, materialize matching bytes in the isolated checkout. Use the same representation for every candidate and independent check; a clean Git status alone does not establish byte identity.
6. Measure the baseline and run the declared checks in isolation. Record commands, exit codes, values and evidence paths. An invalid baseline or required-check failure holds optimization until resolved; do not silently waive the check.