llm-as-judge-evaluation

LLM-as-Judge Evaluation

When to Use This Skill

Use LLM-as-Judge evaluation when you need to:

• Compare a fine-tuned model vs frontier — Does the student beat the teacher on your task?
• Quality gates before deployment — Automated go/no-go on model releases
• Continuous evaluation — Monitor production model quality over time
• Generate preference data — Create (chosen, rejected) pairs for DPO/RLHF training
• Evaluate without ground truth — When exact answers don't exist (creative, open-ended tasks)

When NOT to Use

• Tasks with verifiable answers (math, code execution) — use exact match or unit tests
• Extremely simple classification — use accuracy/F1 directly
• Safety evaluation — use dedicated safety benchmarks, not general judges

Pairwise Comparison

The most reliable LLM-as-judge method. Show a judge two outputs (A and B) and ask which is better.

Basic Implementation

import openai
import json
import random

client = openai.OpenAI()

PAIRWISE_PROMPT = """You are an expert evaluator. Compare two responses to the same prompt.

## Task Context
{task_description}

## User Input
{user_input}

## Response A
{response_a}

## Response B
{response_b}

## Evaluation Criteria
{criteria}

Which response is better? Consider all criteria above.
Return JSON: {{"winner": "A" or "B" or "tie", "reasoning": "brief explanation"}}"""


def pairwise_compare(user_input, response_a, response_b, task_description, criteria,
                     model="gpt-4o", swap_positions=True):
    """Compare two responses with position bias mitigation."""
    results = []

    # First comparison: A=position1, B=position2
    prompt = PAIRWISE_PROMPT.format(
        task_description=task_description,
        user_input=user_input,
        response_a=response_a,
        response_b=response_b,
        criteria=criteria,
    )
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    result1 = json.loads(resp.choices[0].message.content)
    results.append(result1["winner"])

    if swap_positions:
        # Second comparison: swap positions to detect position bias
        prompt_swapped = PAIRWISE_PROMPT.format(
            task_description=task_description,
            user_input=user_input,
            response_a=response_b,  # Swapped
            response_b=response_a,  # Swapped
            criteria=criteria,
        )
        resp2 = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt_swapped}],
            response_format={"type": "json_object"},
            temperature=0,
        )
        result2 = json.loads(resp2.choices[0].message.content)
        # Reverse the swapped result
        swapped_winner = {"A": "B", "B": "A", "tie": "tie"}[result2["winner"]]
        results.append(swapped_winner)

    # Aggregate: both must agree, otherwise tie
    if len(set(results)) == 1:
        return results[0]
    return "tie"