benchmark-writer

Benchmark Writer

What is the benchmark?

Benchmark = scalar evaluation function. Takes codebase state, outputs ONE number. Tells orchestrator how far from done.

NOT a plan. NOT exit conditions. NOT a to-do list. MEASUREMENT INSTRUMENT. Same benchmark runs every iteration, produces comparable score. Trajectory (down for MINIMIZE, up for MAXIMIZE) reveals progress.

Belongs in benchmark: score formula, programmatic checks, data science metrics (MSE, F1, correlation), binary checklist items, fuzzy scales (0-10 with rubrics), iteration log.

Does NOT belong (MOST COMMON MISTAKE): exit conditions, completion conditions, convergence criteria. ALWAYS PROGRAM.md, NEVER BENCHMARK.md. Writing "stop", "exit", "completion", "converge" in benchmark - STOP, move to program.

Prerequisites

PROGRAM.md exists, user-approved.

Process

Round 1: Identify measurable signals

Read PROGRAM.md. ASK user - all in ONE message:

1. What can we measure programmatically? Propose concrete metrics per work item:

   • Line counts (wc -l), function counts (grep -c "def ")
   • Test counts (pytest --co -q | tail -1), test pass rate
   • Lint violations (ruff check --statistics)
   • Complexity scores (radon cc -s -a)
   • File existence (test -f path)
   • grep pattern counts
   • Custom script output (one-liner computing a metric)
   • Data science metrics (for models, simulations, statistical behavior):
     • Error: MSE, RMSE, MAE, MAPE
     • Distribution: KL divergence, Wasserstein, Kolmogorov-Smirnov
     • Classification: F1, precision, recall, accuracy, ROC-AUC
     • Correlation: Pearson r, Spearman rho, R-squared
     • Statistical: p-values, chi-squared, t-test
     • Custom: any domain metric from simulation/model output

2. Target per metric? Current → target.

3. What can't be measured programmatically? Becomes fuzzy scale (0-10) with rubric. Last resort. Every fuzzy scale justifies why.

4. Execution recipe per check? Exact command, script, procedure. Repeatable:

   • Shell: make test, pytest --co -q | tail -1, ruff check --statistics | tail -1
   • Python: one-liners or dedicated scripts outputting a number
   • Scenario tests:
     • Playwright: npx playwright test --reporter=json | jq '.stats.unexpected'
     • API: curl -s endpoint | jq '.status' or pytest fixture
     • Simulation: python run_simulation.py --config test.yaml | grep 'metric:'
     • Generative: prompt template + expected output pattern (e.g. "run claude -p 'prompt', check output contains X")
   • Baselines: store baseline for before/after comparisons