llm-as-judge

LLM-as-Judge

Use a strong LLM to evaluate another LLM's output. Done right, it's fast, cheap, and correlates with human judgment. Done wrong, it's biased, inconsistent, and misleading.

When to Use

• Scaling eval beyond what humans can review
• Measuring open-ended outputs (summaries, code quality, helpfulness) where rule-based metrics fail
• Pairwise model comparison (A vs B on the same input)
• CI checks on agent outputs

When NOT to Use

• High-stakes decisions (medical, legal) — need humans
• When the judge is the same model as the generator — biased toward its own style
• Very short outputs where a rule can decide — exact_match is cheaper
• Tasks the judge can't do itself — if it can't write good code, it can't judge code well

Three Common Patterns

1. Rubric Scoring

Judge rates one output against explicit criteria on a 1-5 scale.

const prompt = `You are evaluating a response. Rate it 1-5 on each criterion.

<user_query>${query}</user_query>
<response>${response}</response>

Criteria:
- accuracy: factually correct?
- helpfulness: addresses what the user asked?
- conciseness: no unnecessary verbosity?

Return JSON: {"accuracy": N, "helpfulness": N, "conciseness": N, "reasoning": "..."}`;

const judgment = await client.messages.create({
  model: "claude-opus-4-6",
  max_tokens: 500,
  messages: [{ role: "user", content: prompt }],
});

Use a stronger model as judge than the one you're evaluating. Opus judges Sonnet; Sonnet judges Haiku.

2. Pairwise Comparison

Show two outputs, judge picks which is better. Most reliable pattern.

const prompt = `Compare two responses to the same query. Pick which is better overall.

<query>${query}</query>
<response_A>${responseA}</response_A>
<response_B>${responseB}</response_B>

Return JSON: {"winner": "A" | "B" | "tie", "reasoning": "..."}`;

To control for position bias, run each pair TWICE with order swapped. Average the judgments.

3. Reference-Based

Compare output to a gold-standard reference: