experiment

Experiment — no claim without a measurement

Optimization needs numbers; null results get recorded too.

1. Define the metric before touching code

Choose one command that prints the number: runtime, RSS, latency, size, or count. Decide now what improvement would count, before seeing results. If no metric command exists, build it first.

2. Baseline

Run at least 3 times and record all values, not only the mean. Pin what you can: seed, input, cache state, and environment; state what remains uncontrolled. Correctness verify must be green before and after.

3. One variable per experiment

Change one thing. Multiple changes make the result uninterpretable.

4. Measure again

Use the Same command, same repetitions, same conditions. A result must beat the baseline spread; Inside the noise = null result.

5. Verdict and record

Follow the repository's existing experiment log; otherwise create and append to docs/experiments.md:

## <date> — <hypothesis>
change: <one variable>
baseline: <all values> | result: <all values>
verdict: WIN <n%> | NULL (inside noise) | LOSS
decision: <kept or reverted + reason>

Losers and nulls are reverted immediately; the log preserves the finding, not the bad diff. A null result is a finding—record it so it is not retried without new evidence. Never delete a previous entry; append a correction when later evidence overturns it. Route load-bearing nulls through /retro.