rag-eval

RAG Eval

Evaluate RAG systems by diagnosing retrieval and generation separately. A single "RAG accuracy" number hides whether the problem is finding the right documents or using them correctly. This skill separates them so you know what to fix.

When to Activate

• User has a RAG pipeline (retriever + generator) with traces
• User wants to know if their RAG system hallucinates
• User is optimizing chunking strategy and needs before/after comparison
• User wants to build a RAG evaluation dataset

Checklist

You MUST create a task for each item and complete them in order:

1. Load RAG traces — validate query + context + answer triples
2. Separate retrieval vs generation — determine which layers to evaluate
3. Run faithfulness evaluation — is the answer grounded in retrieved docs?
4. Run retrieval evaluation — are the right documents retrieved?
5. Build diagnostic matrix — cross-tabulate to find root cause
6. Output findings — prioritized issues with concrete fixes

Fast path: run the bundled script

Once each trace has a faithfulness judgment (and ideally a retrieval signal), build the retrieval-vs-generation diagnostic matrix with the bundled, tested script (scripts/rag_diagnostic.py, standard library only, no OpenJudge dependency):

python scripts/rag_diagnostic.py --traces traces.jsonl

Trace rows: {"faithful":bool} or {"faithfulness_score":1-5} (>= 4 = faithful), plus an optional retrieval signal {"retrieval_good":bool} or {"recall_at_k":0-1} (>= 0.5 = good). It prints the generation faithful/hallucinating split, the 2×2 matrix when a retrieval signal is present, and the primary issue (retrieval vs generation). --self-test to verify it.

Steps below explain how to separate the layers and produce the faithfulness/retrieval signals (with OpenJudge graders or any judge).

Step 1: Load RAG Traces

The minimum data needed per trace:

# Each trace must contain:
trace = {
    "query": "What is the return policy?",
    # retrieved_docs are dicts with id + text (the id is required for retrieval
    # metrics like Recall@k; the text is required for the faithfulness check).
    "retrieved_docs": [
        {"id": "doc_1", "text": "Returns are accepted within 30 days..."},
        {"id": "doc_2", "text": "Refunds are issued to the original..."},
    ],
    "answer": "You can return items within 30 days for a full refund.",
    "reference_answer": "Our 30-day return policy allows full refunds.",  # optional
    "gold_doc_ids": ["doc_1", "doc_3"],  # optional — which docs should have been retrieved
}