run-benchmark

Run Benchmark

HARD GATE — Do NOT use benchmark scores to declare a skill "good" or "bad" in isolation. Benchmarks measure relative quality vs. a baseline — they catch regressions, they do not certify correctness.

Reads benchmark definitions from specs/benchmarks/, executes each scenario's grader with and without the skill loaded, and writes a structured pass@k report with delta grading that evolve-skill consumes.

With/Without-Skill Delta Grading

Every scenario runs N times (default 3) in two modes: with the skill loaded and without (bare agent with only CLAUDE.md). The delta Δ = pass@k_with − pass@k_without isolates the skill's causal contribution. A negative delta is a regression flag.

Train/Validation Split

Benchmark definitions partition scenarios into two sets:

Set	Tag	Purpose
Train	split: train	Development scenarios — used while iterating. Hitting 100% on train is expected.
Validation	split: validation	Held-out scenarios — the real quality signal. Overfitting train while validation stagnates is a design smell.

pass@k is reported separately for train and validation. Validation score is authoritative; train score is iteration guidance only.

Usage

bash scripts/run-benchmark.sh <skill-name>           # benchmark single skill
bash scripts/run-benchmark.sh --all                  # benchmark all with definitions
bash scripts/run-benchmark.sh <skill-name> --baseline # pin results as baseline

Process

1. Locate definition — Read specs/benchmarks/<skill>.yaml. If absent, stop with message.

2. Partition scenarios — Split by split field (train → iteration, validation → authoritative, default: validation).

3. Run each scenario (N-run delta) — For each scenario, run grader N times (default 3, configurable via runs:):

   • Without skill: Agent with only CLAUDE.md/CONVENTIONS.md
   • With skill: Agent with the skill under test active
   • Code grader: bash -c <command>, exit 0 → PASS. Timeout: 15s.
   • Rubric grader: yes/no per criterion, ≥ 80% yes → PASS.
   • Record: {scenario_id: {with: [P/F,...], without: [P/F,...]}}