llm-evaluation

QUARANTINE — Community tier. Patterns here are unvalidated. Read-only tools only. Verify against official docs (Anthropic, LangSmith) before applying to production.

LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

When to Use This Skill

• Measuring LLM application performance systematically
• Comparing different models or prompts
• Detecting performance regressions before deployment
• Validating improvements from prompt changes
• Building confidence in production systems
• Establishing baselines and tracking progress over time
• Debugging unexpected model behavior

Core Evaluation Types

1. Automated Metrics

Text Generation:

• BLEU: N-gram overlap (translation)
• ROUGE: Recall-oriented (summarization)
• METEOR: Semantic similarity
• BERTScore: Embedding-based similarity
• Perplexity: Language model confidence

Classification:

• Accuracy, Precision/Recall/F1, Confusion Matrix, AUC-ROC

Retrieval (RAG):

• MRR, NDCG, Precision@K, Recall@K

2. LLM-as-Judge

Use stronger LLMs to evaluate weaker model outputs.

Approaches:

• Pointwise: Score individual responses
• Pairwise: Compare two responses (A/B)
• Reference-based: Compare to gold standard
• Reference-free: Judge without ground truth

from anthropic import Anthropic
from pydantic import BaseModel, Field
import json

class QualityRating(BaseModel):
    accuracy: int = Field(ge=1, le=10)
    helpfulness: int = Field(ge=1, le=10)
    clarity: int = Field(ge=1, le=10)
    reasoning: str

async def llm_judge_quality(response: str, question: str, context: str = None) -> QualityRating:
    client = Anthropic()
    prompt = f"""Rate the following response:
Question: {question}
{f'Context: {context}' if context else ''}
Response: {response}

Provide ratings in JSON: {{"accuracy": <1-10>, "helpfulness": <1-10>, "clarity": <1-10>, "reasoning": "<explanation>"}}"""
    message = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=500,
        system="You are an expert evaluator of AI responses.",
        messages=[{"role": "user", "content": prompt}]
    )
    return QualityRating(**json.loads(message.content[0].text))