advanced-evaluation

Advanced Evaluation: LLM-as-a-Judge Techniques

Master the use of language models as evaluators for subjective quality dimensions that deterministic checks cannot capture. LLM-as-judge evaluation complements assertion-based testing by assessing coherence, helpfulness, safety, instruction-following, and other qualities that require semantic understanding. The core challenge is that evaluators are themselves non-deterministic — they exhibit systematic biases that must be detected and mitigated.

When to Activate

Activate this skill when:

• Implementing LLM-based scoring for agent outputs
• Designing pairwise comparison evaluations
• Building rubrics for subjective quality assessment
• Mitigating evaluator biases (position, verbosity, self-preference)
• Calibrating evaluator models against human judgments
• Measuring inter-evaluator reliability

Do not activate this skill for adjacent work owned by other skills:

• Deterministic checks, regression testing, or metric design: evaluation.
• Designing autonomous agent loops with evaluation gates: harness-engineering.
• Choosing evaluation strategy at the project level: project-development.

Core Concepts

LLM-as-judge evaluation uses a language model to score or compare outputs against quality criteria. Three evaluation modes serve different purposes:

1. Direct scoring — A single evaluator rates an output on a scale (1-5, 1-10). Simple but susceptible to calibration drift.
2. Pairwise comparison — An evaluator chooses between two outputs. More reliable than direct scoring because relative judgments are easier than absolute ones.
3. Reference-based grading — An evaluator compares output against a gold-standard reference. Most reliable but requires reference answers.

All three modes are subject to systematic biases: position bias (preferring the first or last option), verbosity bias (preferring longer responses), self-preference bias (preferring outputs from the same model family), and anchoring bias (being influenced by the scoring scale presentation).