agent-eval-design

Agent Evaluation & Benchmark Design

Design evaluations for AI agents, dev tools, retrieval, and repo-scale automation. Evaluation quality beats benchmark size — a small set of real, uncontaminated, decision-driving tasks is worth more than thousands of synthetic puzzles.

Core principles

• Measure real tasks. Representative workloads over synthetic puzzles. If no one does the task in practice, the number is noise.
• Separate capability from prompt engineering. Hold the harness/prompt fixed when comparing models; hold the model fixed when comparing prompts. Report which you varied. A gain you can't attribute is a gain you can't ship.
• Every metric must change an engineering decision. Before adding a metric, name the decision it informs. If nothing changes based on its value, cut it.
• Reproducibility is a first-class result. Pin model versions, seeds, dataset hashes, harness commit, and date. An unrepeatable eval is an anecdote.
• Minimize contamination. Assume public benchmarks are in training data; prefer held-out, private, or post-cutoff tasks and say so.

Dimensions to consider

Correctness · completeness · reliability · latency · cost · determinism · reproducibility · developer effort · failure recovery · robustness. Pick the few that map to real decisions for this system; don't report all ten by reflex.

Repository-scale evaluations

For agents that operate over codebases, evaluate the axes that synthetic tasks miss: repository understanding, cross-file reasoning, architectural consistency, migration quality, semantic correctness (not just diff-match), dependency propagation, test generation, and documentation accuracy. Verify outcomes by execution (tests pass, build green, behavior preserved) rather than string similarity to a reference solution.

Benchmark design — audit before trusting

Before believing a benchmark, check it for:

• Contamination / dataset leakage — is the answer reachable from training data or from the prompt itself?
• Unrealistic tasks — puzzle-shaped work no engineer actually does.
• Missing edge cases — the failure modes that matter live in the tail.
• Insufficient statistical power — enough trials and items to distinguish signal from run-to-run variance? Report variance/CIs, not a single point.
• Evaluation blind spots — what the metric structurally cannot see (e.g. pass@1 hides flakiness; exact-match hides correct-but-different solutions).