agent-evaluation

Agent Evaluation

Decide whether a stochastic candidate (prompt, model, retrieval, or tool agent) beats its baseline under realistic nondeterminism. quality-assurance owns deterministic software tests, prompt-design prepares the candidate, and production-readiness decides operability.

Workflow

1. State the decision: name the evaluated unit, baseline, candidate, segments, primary metric, guardrails, and kill criterion.
2. Pick the mode: development mode (frozen cases, paired runs, deterministic graders) returns only ITERATE or INCONCLUSIVE; release or adoption mode adds a sealed holdout, a predeclared decision rule, blinded graders, and adequate repetitions.
3. Pin candidate identity: model provider and immutable version, prompts or their hashes, tool schemas, retrieval snapshot, code revision, sampling, retries, and grader versions; a model name alone is not identity.
4. Build scenario sets: a versioned development corpus from sanitized logs, incidents, and known failures that covers success paths, adversarial and refusal cases, tool errors, and long-tail segments; release mode adds a sealed, contamination-checked holdout.
5. Capture traces: final response plus tool calls, arguments, results, denials, retrieval sources, errors, tokens, latency, cost, and cleanup; grade attempted unsafe actions even when the gateway blocked them.
6. Stack graders: deterministic schema checks, executable tests, and trace assertions first; pairwise or criteria scoring for semantic dimensions; a blinded pinned model judge only where code cannot decide; humans for safety and disagreement.
7. Calibrate and blind: test graders on labeled passes, failures, and edge cases; give judges randomized opaque candidate labels, strip provider metadata, swap order, and report false positives, false negatives, and blind spots. A candidate never grades itself.
8. Justify trial counts: choose trials, seeds, minimum detectable difference, and uncertainty method before running, per the evaluation protocol; return INCONCLUSIVE when the count cannot separate the threshold.
9. Run paired comparisons: baseline and candidate on identical cases, tools, budgets, and fresh state; retain every valid run, failure, timeout, and grader disagreement.
10. Analyze by risk and segment: report capability, reliability, safety, latency, tokens, and cost separately; an aggregate gain never hides a safety regression or a minority-segment loss.
11. Spend the holdout once: Freeze the decision rule before the sealed holdout, keep an append-only exposure log, run the frozen candidate on it once, record the corpus hash, and rotate an exposed holdout.
12. Make the call: return ADOPT, ITERATE, REJECT, or INCONCLUSIVE tied to the predeclared thresholds, with the cheapest next evidence, in the evaluation brief shape.