methodology

LLM evaluation methodology

This page states the rules; answer from it. It runs, scores, scaffolds and edits nothing.

• Pointer: when a specific must be current, fetch Define your success criteria and Build evaluations live; the evals cookbook is anthropics/claude-cookbooks misc/building_evals.ipynb.
• As of: 2026-08-08 (distilled); pointer anchors verified 2026-10-11
• Recheck trigger: either section is retitled, moves, or changes the guidance these rules rest on.

Background files

Each is background for a human reader, with its own source stamp. For runnable code, or a specific that must be current, check the source page instead.

Quick decision guide

• "Where do I start?" For a Claude API app, with /claude-api build-eval (see Route by repository kind). Otherwise, define measurable success criteria first; evals test against them; only then iterate on prompts.
• "Is this criterion good?" It names a specific quality, a number or defined scale, a realistic target, and ties to a user need. The target number is realistic only when a measured baseline, a prior result, a benchmark, or expert review justifies it; how severe a miss would be, or that a grader can check the number, does not. "Good performance" fails all four.
• "Rewrite this criterion" The rewrite carries, inline, a number or defined scale, the set of trials it is measured over, and a target grounded the way the Achievable property allows. When the user gives no data, state the target as a quantity relative to the current baseline (for example "at most half the current rate") over a trial set of stated size (for example "500 conversations"), without inventing the baseline's value; "lower than the baseline" is a direction, not a target. Never present a made-up baseline figure or expert agreement as fact. X/Y/Z placeholders and "set the target later" are not a usable criterion.
• "Which grading method?" The fastest, most reliable, most scalable that fits: code-based if the output can be constrained to allow it; LLM-graded for judgment; human only as a last resort.
• "Can I automate this seemingly subjective eval?" Usually. Constrain the output format, reformat to multiple choice, or use an LLM grader with a tight rubric and constrained verdict.
• "How many cases?" Prefer volume with automated grading over a few hand-graded showpieces; generate more from a baseline set with Claude, human-reviewed. Volume means cheaper grading, never easier cases.
• "Can I trust my LLM grader?" Only after reading samples of its verdicts against your own judgment; and grade with a different model than the one that generated the output.
• "How do I grade an agent?" On the environment state it left, with a grader it cannot reach; see Agent evals: grader hygiene.
• "One metric or several?" Several. Most use cases need multidimensional criteria (fidelity, safety, latency, cost); a single headline metric hides regressions.