rag-evaluate-quality

rag-evaluate-quality — measure, do not guess

Use this workflow when the user asks to evaluate retrieval, after a relevant change, or as part of an already authorized recurring evaluation. Do not infer that every session needs a benchmark or that invoking this skill creates a recurring schedule.

Workflow

1. Call get_index_stats(). The response contains a stats object. Record stats.total_documents, stats.total_chunks, stats.embedding_model, stats.embedding_dim, and stats.query_cache.hit_rate. The cache rate describes repeated-query reuse, not relevance.

2. Reuse an independently selected evaluation set, or prepare questions with known answer documents. Each case has one non-empty question and one non-empty expected path:

   [
     {"query": "authentication design", "expected_filepath": "docs/adr/0018-auth.md"},
     {"query": "retry policy", "expected_filepath": "docs/adr/0031-retries.md"}
   ]
   

   These are illustrative paths. Verify that expected documents exist in the actual corpus. Do not derive the expected answer from whichever source the current search happens to rank first.

3. Serialize the array as a JSON string and pass the test_cases parameter:

   import json
   
   cases = [
       {"query": "authentication design", "expected_filepath": "docs/adr/0018-auth.md"},
       {"query": "retry policy", "expected_filepath": "docs/adr/0031-retries.md"},
   ]
   evaluate_retrieval(test_cases=json.dumps(cases))
   

   The MCP tool returns mrr_at_5, recall_at_5, total_queries, and per_query. It does not return Precision@5. Invalid cases are rejected before searches execute.

4. Inspect individual misses and rank changes before interpreting aggregates:

   Metric	Meaning
   MRR@5	Mean reciprocal rank of the expected document; a miss contributes zero
   Recall@5	Fraction of cases whose expected document occurs in the first five results
   found_at_rank	Rank for each case, or null if the expected document was not found