agent-evaluation

Agent Evaluation

Build evidence that can inform a release owner, not a showcase of favorable examples or a safety certification.

Use when

• Define quality before building or changing an agent.
• Compare prompts, models, tools, memory strategies, or orchestration patterns.
• Convert production failures into regression cases.
• Establish a repeatable release gate or human-review plan.

Inputs

Collect the agent objective, users, supported tasks, unacceptable outcomes, current baseline, execution environment, available traces, and evaluation budget. State assumptions when an input is unavailable.

Output contract

Produce:

1. An evaluation brief with scope, risks, hypotheses, and frozen system versions.
2. A dataset manifest with provenance, categories, splits, and contamination controls.
3. A scoring specification with rubrics, graders, thresholds, and tie-breaking rules.
4. Reproducible run settings, aggregate results, uncertainty, and baseline deltas.
5. A failure taxonomy, representative cases, evidence limits, and a decision memo for the accountable release owner.

Workflow

1. Define the unit under test and the decision the evaluation must support. Separate model quality from tool, retrieval, policy, and infrastructure failures.
2. Convert user goals and risks into observable criteria. Include task success, safety, latency, cost, and escalation quality only when relevant.
3. Build representative cases from real distributions where permitted. Add boundary, long-tail, malformed-input, tool-failure, and adversarial cases. Keep a holdout set isolated from prompt iteration.
4. Select the least subjective reliable grader. Prefer deterministic checks for structured facts, rubric-bound model graders for semantic quality, and blinded human review for high-impact or disputed cases. Read evaluation-patterns.md when selecting graders or gates.
5. Freeze prompts, model versions, tools, data snapshots, seeds when supported, retries, and timeouts. Run the candidate and baseline under equivalent conditions; repeat stochastic cases.
6. Inspect case-level failures before trusting aggregates. Slice results by task, risk, language, tool, and user cohort where sample sizes permit.
7. Set a release gate that combines minimum critical-case performance, non-regression against baseline, and operational limits. Label underpowered results as inconclusive.
8. Save failed production-like cases as regression fixtures without exposing private data.