evaluate-rag

Evaluate RAG

Overview

1. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both.
2. Build a retrieval evaluation dataset: queries paired with relevant document chunks.
3. Measure retrieval quality with Recall@k (most important for first-pass retrieval).
4. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?).
5. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation.

Prerequisites

Complete error analysis on RAG pipeline traces before selecting metrics. Inspect what was retrieved vs. what the model needed. Determine whether the problem is retrieval, generation, or both. Fix retrieval first.

Core Instructions

Evaluate Retrieval and Generation Separately

Measure each component independently. Use the appropriate metric for each retrieval stage:

• First-pass retrieval: Optimize for Recall@k. Include all relevant documents, even at the cost of noise.
• Reranking: Optimize for Precision@k, MRR, or NDCG@k. Rank the most relevant documents first.

Building a Retrieval Evaluation Dataset

You need queries paired with ground-truth relevant document chunks.

Manual curation (highest quality): Write realistic questions and map each to the exact chunk(s) containing the answer.

Synthetic QA generation (scalable): For each document chunk, prompt an LLM to extract a fact and generate a question answerable only from that fact.

Synthetic QA prompt template:

Given a chunk of text, extract a specific, self-contained fact from it.
Then write a question that is directly and unambiguously answered
by that fact alone.

Return output in JSON format:
{ "fact": "...", "question": "..." }

Chunk: "{text_chunk}"

Adversarial question generation: Create harder queries that resemble content in multiple chunks but are only answered by one.

Process:

1. Select target chunk A containing a clear fact.
2. Find similar chunks B, C using embedding search (chunks that share terminology but lack the answer).
3. Prompt the LLM to write a question using terminology from B and C that only chunk A answers.