eval-llm-judge

LLM Judge Evaluator Skill

An LLM-as-judge evaluator uses a language model to score another model's output against a defined criterion. It handles semantic properties that code-based checks cannot: is the ranking defensible? Did the agent handle ambiguity gracefully? Does the output align with the reference?

The tradeoff: judges are slower, more expensive, and non-deterministic. They require calibration against human labels before you can trust their scores. Use them only after code-based checks have covered the structural requirements.

When to use this skill

Write an LLM judge when the property requires reasoning that a human reviewer would apply:

• Output correctness: are the results correct and in a defensible order or format?
• Edge case handling: did the agent appropriately flag ambiguity or decline gracefully?
• Reference alignment: does the output agree with a gold-standard expected response?
• Tone and framing: is the hedging appropriate for a speculative or high-stakes claim?
• Faithfulness: does the output stay within what the cited sources actually say?

Do NOT write an LLM judge for:

• Properties a code check can handle (count, format, presence) — these are cheaper and faster
• Compound criteria — write one judge per criterion, not one judge that checks everything

---

Workflow

Step 1 — Find the project's scorer contract and judge helper

Before drafting, look for where the project already keeps its evaluators: a scorers/ or evals/ folder, a types or contract file, existing judges, a README that says how they run. If there is one, write the judge in that language and follow the contract exactly: the function signature, the return shape, the file name and folder. If the project has a helper that calls the judge model, use it rather than the runner template below, so every judge shares one model, one output schema and one way of handling errors.

For example, Mapper's course scorers live in scorers/course/, one TypeScript file per scorer. Each judge exports its PROMPT_TEMPLATE and default-exports a function that fills the template and calls judge(name, prompt) from scorers/course/types.ts. That helper asks Gemini 3.5 Flash-Lite, at temperature 0, for {critique, score} and returns { name, score: 0 | 1, metadata: { reason } }, with the critique as the reason.