experimentation-loop

Experimentation Loop

Purpose: Drive a measurable metric in a chosen direction by running many small, isolated experiments, keeping wins, and reverting losses. Scope: Branch isolation, metric definition, attempt audit trail, keep/revert decision rule.

---

When to Use This Skill

• A target metric exists and can be measured by a deterministic command
• The desired direction is known (lower latency, higher pass-rate, smaller bundle)
• Many small attempts are likely needed
• Reverting a bad attempt is cheap (single git checkout)
• A durable record of every attempt is valuable

When NOT to Use

• Correctness work where the goal is binary "tests pass" -- use iterative-loop
• Tasks without a measurable metric or a way to compute it from a single command
• Risk-bearing changes that should not auto-revert (use a normal review flow)
• Production hotfixes (use targeted change with explicit review)

Prerequisites

• Clean working tree on a non-main branch
• A metric command that exits 0 and prints a single numeric value, OR a JSON value reachable by a fixed JSON pointer
• A baseline measurement captured before the loop starts
• Permission to commit on the experimentation branch
• The loop runner MUST record files_changed (modified tracked files) and files_added (new untracked files) per attempt; reverts without that list are forbidden

---

Decision Tree

Is the goal a measurable number?
+- No -> use iterative-loop or standard workflow
- Yes
   +- Can the metric be computed by one command?
   |  +- No -> wrap it in a script that prints one number, then proceed
   |  - Yes -> proceed
   - Is each attempt cheap to revert?
      +- No -> do not auto-revert; require human gate per attempt
      - Yes -> use this loop

---

Loop Pattern

1. Capture baseline metric on isolated branch
2. Propose a single, narrow change with a hypothesis
3. Apply the change
4. Run the metric command
5. Compare against current best
   +- Improvement -> commit, update best, log win
   - Regression or no change -> revert working tree, log loss
6. Optional: stop if N consecutive losses or budget exhausted
7. Repeat until target reached, max attempts hit, or diminishing returns