agent-eval-design

Agent Evaluation & Benchmark Design

Design evaluations for AI agents, dev tools, retrieval, and repo-scale automation. Evaluation quality beats benchmark size — a small set of real, uncontaminated, decision-driving tasks is worth more than thousands of synthetic puzzles.

Core principles

• Measure real tasks. Representative workloads over synthetic puzzles. If no one does the task in practice, the number is noise.
• Separate capability from prompt engineering. Hold the harness/prompt fixed when comparing models; hold the model fixed when comparing prompts. Report which you varied. A gain you can't attribute is a gain you can't ship.
• Every metric must change an engineering decision. Before adding a metric, name the decision it informs. If nothing changes based on its value, cut it.
• Reproducibility is a first-class result. Pin model versions, seeds, dataset hashes, harness commit, and date. An unrepeatable eval is an anecdote.
• Minimize contamination. Assume public benchmarks are in training data; prefer held-out, private, or post-cutoff tasks and say so.

Dimensions to consider

Correctness · completeness · reliability · latency · cost · determinism · reproducibility · developer effort · failure recovery · robustness. Pick the few that map to real decisions for this system; don't report all ten by reflex.

From traces to a regression suite

Use the stages that match the available evidence; an existing, validated rubric does not need to be rediscovered on every run.

1. Discover failures. Review real inputs, outputs, tool results, and outcomes before selecting product-quality metrics. Read trace review when failure modes are unclear. Supplement observations with explicit requirements and adversarial cases; absence from sampled traces does not make a requirement unnecessary.
2. Define the rubric. For each failure mode, record evidence, the expected behavior, valid alternatives, and the decision a check would inform. Resolve ambiguous requirements before treating disagreement as model failure.
3. Validate the evaluator. Prefer execution or deterministic checks where they actually measure the criterion. Use a narrow LLM judge for semantic judgments and read judge calibration before trusting its scores. Deterministic verifiers also need valid controls, independent references, and evidence that legitimate solutions pass.
4. Design regression coverage. Turn confirmed failures into regression cases, retain representative held-out measurement data, and compare matched cases across changes. Report targeted-suite results separately from population estimates. Revisit failure categories and evaluator validity as behavior drifts.