llm-evaluator

LLM Evaluator

A framework for measuring LLM output quality — beyond vibes, before production.

When to Use

• "Is Model A better than Model B for my use case?"
• "Did my prompt change improve quality?"
• "My LLM outputs look worse after an update — prove it"
• Before deploying any LLM-powered feature

The Three Evaluation Approaches

Use all three. Each catches different problems.

1. LLM-as-Judge

Fast, scalable, good signal. Use a stronger model to evaluate a weaker one's outputs.

JUDGE_PROMPT = """You are evaluating an AI assistant's response quality.

Rate the response on each criterion from 1-5:
- Accuracy: Is the information correct and factual?
- Completeness: Does it address all parts of the question?
- Clarity: Is it easy to understand?
- Conciseness: Does it avoid unnecessary verbosity?

Question: {question}
Response to evaluate: {response}
Reference answer (if available): {reference}

Return JSON only:
{{"accuracy": N, "completeness": N, "clarity": N, "conciseness": N, "reasoning": "brief explanation"}}
"""

async def judge_response(question: str, response: str, reference: str = "") -> dict:
    result = await judge_model.ainvoke(
        JUDGE_PROMPT.format(question=question, response=response, reference=reference)
    )
    return json.loads(result.content)

Known biases: LLM judges favor longer answers, prefer their own model's style, and are inconsistent on borderline cases. Mitigate by averaging across 3 judge calls and using a different model family as judge.

2. Automated Metrics

For structured tasks with clear correct answers:

from rouge_score import rouge_scorer
from nltk.translate.bleu_score import sentence_bleu

def compute_metrics(prediction: str, reference: str) -> dict:
    # ROUGE — for summarization
    scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"])
    rouge = scorer.score(reference, prediction)

    # Exact match — for extraction tasks
    exact = prediction.strip().lower() == reference.strip().lower()

    # F1 over tokens — for QA
    pred_tokens = set(prediction.lower().split())
    ref_tokens = set(reference.lower().split())
    common = pred_tokens & ref_tokens
    precision = len(common) / len(pred_tokens) if pred_tokens else 0
    recall = len(common) / len(ref_tokens) if ref_tokens else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0

    return {
        "rouge_l": rouge["rougeL"].fmeasure,
        "exact_match": exact,
        "token_f1": f1,
    }