llm-evaluation

LLM Evaluation & Testing

Comprehensive guide to evaluating and testing LLM applications including prompt testing, output validation, hallucination detection, benchmark creation, A/B testing, and quality metrics.

---

Quick Reference

When to use this skill:

• Testing LLM application outputs
• Validating prompt quality and consistency
• Detecting hallucinations and factual errors
• Creating evaluation benchmarks
• A/B testing prompts or models
• Implementing continuous evaluation (CI/CD)
• Measuring retrieval quality (for RAG)
• Debugging unexpected LLM behavior

Metrics covered:

• Traditional: BLEU, ROUGE, BERTScore, Perplexity
• LLM-as-Judge: GPT-4 evaluation, rubric-based scoring
• Task-specific: Exact match, F1, accuracy, recall
• Quality: Toxicity, bias, coherence, relevance

---

Part 1: Evaluation Fundamentals

The LLM Evaluation Challenge

Why LLM evaluation is hard:

1. Subjective quality - "Good" output varies by use case
2. No single ground truth - Multiple valid answers
3. Context-dependent - Same output good/bad in different scenarios
4. Expensive to label - Human evaluation doesn't scale
5. Adversarial brittleness - Small prompt changes = large output changes

Solution: Multi-layered evaluation

Layer 1: Automated Metrics (fast, scalable)
  ↓
Layer 2: LLM-as-Judge (flexible, nuanced)
  ↓
Layer 3: Human Review (gold standard, expensive)

Evaluation Dataset Structure

from dataclasses import dataclass
from typing import List, Optional

@dataclass
class EvalExample:
    """Single evaluation example."""
    input: str  # User input / prompt
    expected_output: Optional[str]  # Gold standard (if exists)
    context: Optional[str]  # Additional context (for RAG)
    metadata: dict  # Category, difficulty, etc.

@dataclass
class EvalResult:
    """Evaluation result for one example."""
    example_id: str
    actual_output: str
    scores: dict  # {'metric_name': score}
    passed: bool
    failure_reason: Optional[str]

# Example dataset
eval_dataset = [
    EvalExample(
        input="What is the capital of France?",
        expected_output="Paris",
        context=None,
        metadata={'category': 'factual', 'difficulty': 'easy'}
    ),
    EvalExample(
        input="Explain quantum entanglement",
        expected_output=None,  # No single answer
        context=None,
        metadata={'category': 'explanation', 'difficulty': 'hard'}
    )
]