benchmarking-models

Benchmarking models

The bench runs N candidate models on one research goal and settles them in a cross-model Elo tournament judged by a separate model. It is the tool for "which model should I use for this", and for checking that a change to prompts or config actually helped.

This runs through the CLI, not the MCP tools. It runs a full pipeline per candidate, so it is several times the size of a single session.

Before running

1. Check the backend with co-scientist:doctor and read billing.
2. Size the run. It scales as candidates times --n times --matches, so five candidates is five full generation passes plus every pairing between them.
3. Read the caps correctly. --budget-per-candidate and --judge-budget multiply: five candidates at 3.0 plus a judge at 5.0 is a cap of 20.0. Under subscription those are usage caps in the engine's odometer units, not prices, and nothing is charged; under metered they are real dollars.
4. Get an explicit go-ahead on the size.

Running

Reproduce the paper's comparison:

co-scientist bench "Identify hypotheses about X" \
  --preset paper \
  --judge claude_cli:sonnet

Compare your own candidates:

co-scientist bench "Identify hypotheses about X" \
  -c opus=claude_cli:opus \
  -c sonnet=claude_cli:sonnet \
  -c codex=codex_cli:gpt-5.6-sol \
  --judge claude_cli:sonnet

Both candidates there run on a subscription, so nothing is billed. Add --budget-per-candidate and --judge-budget (USD caps, defaults 3 and 5) only when a candidate or the judge is a metered API model.

Candidate syntax is label=backend:model[@mode]. Mode pipeline is the default and runs the full agent loop. Mode direct is a single raw model call with no tools, which is the honest baseline for "does the multi-agent structure earn the extra calls".

Useful flags:

• --n hypotheses per candidate, default 2
• --matches tournament matches per pair, default 2
• --goldset scores against a known answer key; --preset paper-aml bundles one. Pass none to disable a preset's default.