ai-evaluation

AI Evaluation

Purpose: Systematically measure and validate AI/ML model quality across accuracy, safety, and reliability dimensions.

---

When to Use This Skill

• Designing evaluation frameworks for LLM-based applications
• Implementing automated evaluation pipelines (CI/CD for AI)
• Measuring RAG pipeline quality (retrieval + generation)
• Running benchmarks for model selection or fine-tuning validation
• Establishing quality gates before model deployment
• Evaluating safety, bias, and alignment properties

Prerequisites

• Test dataset with ground truth (or human evaluation plan)
• Access to the model/system under test
• Evaluation metrics selected for the task type

Decision Tree

What are you evaluating?
+- Text generation quality?
|  +- Open-ended? -> Human eval + LLM-as-judge
|  +- Structured output? -> Exact match + schema validation
|  +- Summarization? -> ROUGE + faithfulness + LLM-as-judge
+- RAG pipeline?
|  +- Retrieval quality -> Context relevance, recall, precision
|  +- Generation quality -> Faithfulness, answer relevancy
|  +- End-to-end -> RAGAS framework
+- Classification / extraction?
|  +- Use standard ML metrics (accuracy, F1, precision, recall)
+- Safety / alignment?
|  +- Toxicity detection, jailbreak resistance, bias testing
+- Agent / tool use?
|  +- Tool call accuracy, task completion rate, step efficiency
+- Comparing models?
|  +- Side-by-side with same test set and metrics

---

Evaluation Dimensions

Dimension	What It Measures	Key Metrics
Correctness	Factual accuracy of outputs	Accuracy, F1, exact match
Faithfulness	Grounded in provided context (no hallucination)	Faithfulness score, hallucination rate
Relevance	Output addresses the question asked	Answer relevancy, context relevancy
Coherence	Logical flow and readability	Coherence score, fluency
Safety	Free from harmful, biased, or toxic content	Toxicity rate, bias scores
Robustness	Consistent across paraphrases and edge cases	Variance across perturbations
Latency	Response time	P50, P95, P99 latency
Cost	Token usage and compute cost	Tokens per request, cost per query