operate-benchmark-lab

Operate the benchmark lab

The operator's manual for the whole benchmark/experiment lifecycle a coding agent can drive end to end. Two interfaces over the same sidecar files: the benchmarks MCP server (preferred for agents — same validation code as the hub API) and the CLI verbs. Execution always happens in a separate executor process; the MCP server and hub never run models. Command matrix, artifact map, and daemon details in reference.md; the tool table and agent loop in docs/agent-operator-surface.md.

Resolve CLI

Prefer the installed understudy binary. If it is unavailable inside a repo checkout, run through the package script:

npm run build
node dist/bin.js benchmarks mcp --root <dir>

MCP registration (Claude Code ~/.claude.json → mcpServers): { "understudy-benchmarks": { "type": "stdio", "command": "understudy", "args": ["benchmarks", "mcp"] } } — default root ~/.understudy/benchmarks; add --root per extra directory.

Safety Gates

• Queueing is not executing. queue_run / understudy runs queue only writes a request file. Model rollouts spend gateway money only when an executor picks the request up; say which executor will, before queueing.
• Trust posture, not per-call dialogs. Spend-adjacent shapes (multi-arm or multi-rollout runs, implicit all-task runs, experiment approval/verdict patches) consult the one-time posture in ~/.understudy/trust.json (understudy trust set, levels local_sandbox < bounded_experiments < hosted_ops). At bounded_experiments+ they proceed with a visible one-line notice (arm count, rough cost) — surface that notice to the user, then keep moving. Below that, the guard returns the one action to offer (understudy trust set bounded_experiments); confirm: true after explicit in-chat consent stays a per-call escape hatch. There is NO default spend cap: the posture's allow_spend_usd_per_run is an opt-in generous stop-loss (warn at 1x with a recorded spend_warning; hard stop only at 2x with spend_stop).
• Local arms are machine-aware. On predicted OOM (onboarding profile / memory probe) or a serve failure, the executor runs the arm on the gateway base model and records it — arm_fallback event plus fallback_reason on every row. Report the fallback; never present a fallen-back arm as a local measurement.
• One executor per benchmark dir. Before starting runs execute --watch, check for a live claim (claimed_by on the request, executor_version on events) — a stale watcher built before a feature landed is the classic corruption hazard; new requests carry requires:[...] so old executors skip them with run_unsupported instead of running them bare.
• Reviews and feedback are append-only ledgers; never edit reviews.jsonl/feedback.jsonl lines in place. Generated tasks are born accepted (review-policy default_decision: "accept") — reviews.jsonl carries explicit overrides only. apply_auto_accepts matters only for benchmarks opted into default_decision: "pending", and is itself the explicit user action — invoke it only when the developer asked.
• Honest reporting only: anomaly rows (rollout_timeout, app_replay_unobserved, structural sentinels) are excluded from aggregates but reported, never fabricated as scores. Overlapping CIs are a tie.
• Verifier-only changes regrade, never rerun. When only the verifier group changed (gold, contract, rubric, metric config — a MINOR bump), re-score the existing trajectories; queueing fresh rollouts for a verifier fix wastes gateway money and destroys comparability. Rerun is reserved for env-group (MAJOR) changes.
• Fixture-test every verifier before trusting it. One known-valid result must pass and one plausible-but-wrong result must fail. A verifier that has never rejected a wrong answer has not been tested; do not regrade or calibrate against it.