bench

Activate When

• /godmode:bench, "benchmark", "compare variants", "A/B metric"
• User has a metric and wants a statistically-honest comparison between 2-3 code states
• NOT for optimization loops — use optimize for that. bench measures, never modifies source.

Inputs

Ask once, cache for the session:

• metric_cmd — shell command printing ONE number to stdout (lower-is-better or higher-is-better, user declares direction).
• variants — list of 2-3 entries. Each variant is a record:
  • name — short label (e.g. main, terse_on, feature_branch)
  • prep — EITHER a git ref (git checkout <ref>) OR an inline shell command that puts the repo into the desired state (e.g. export GODMODE_TERSE=1, git checkout feature-branch)
  • teardown — optional shell command to undo prep (default: git checkout - for refs, unset VAR for env)
• baseline — name of the variant all deltas are computed against. Must match one variants[].name.
• N — runs per variant. Default 5. Minimum 3. Maximum 20.
• variance_threshold — stdev/mean ratio that flags a variant noisy. Default 0.05 (5%).

Workflow

1. Validate inputs — metric_cmd must emit a single number; baseline must match a variant; N >= 3.
2. Snapshot starting git state (HEAD sha, branch, dirty bit). Refuse to run if working tree is dirty.
3. FOR each variant in order: a. Run prep. Abort variant on non-zero exit, mark prep_failed. b. Run metric_cmd N times. Collect numbers into an array. c. Compute mean, median, stdev, cv = stdev/mean. d. IF cv > variance_threshold: retry the FULL N-run block up to 3 times (variance recovery). e. IF cv still > threshold after 3 retries: mark variant measurement_error, record best-effort numbers. f. Run teardown. Fail loud if teardown leaves repo dirty.
4. Restore starting git state exactly (same sha, same branch). Verify with git rev-parse HEAD.
5. Compute delta_pct for each variant vs. baseline: (variant.median - baseline.median) / baseline.median * 100.
6. Emit outputs (see Output Format).