retrieval-eval

Retrieval Eval

You measure whether retrieval still works, and when it does not, you name the knob that moved it: chunk size, embedder, or reranker.

Two harnesses answer two different questions. Don't mix them up:

Question	Command	What it measures
"Did anything regress since the baseline?"	gnosis-mcp eval --json	A fixed in-repo fixture
"How does this corpus do?"	python tests/bench/bench_real_corpus.py	The caller's corpus + golden queries

The fixture limitation — state it out loud

gnosis-mcp eval does not read the user's corpus. It creates a throwaway SQLite DB in a temp directory, ingests the sample docs baked into tests/eval/test_search_quality.py, and scores the cases in tests/eval/cases.json at K=5. It is a canary for the ranking code and the harness — not a statement about anyone's docs.

So:

• Never report its numbers as "your corpus scores 0.9". Say "the bundled fixture scores …", or run the corpus harness instead.
• It requires the repository checkout. An installed-only package exits 1 with a message that tests/ is required.
• It reports Hit@K, MRR and mean Precision@K. It does not report nDCG@10 — that only comes from the corpus harness or BEIR.

Actual --json keys: cases, hit_rate_at_k, mrr, mean_precision_at_k, k.

Workflow

1. Locate the current numbers

gnosis-mcp eval --json

Then compare against the saved baseline at ~/.local/share/gnosis-mcp/eval-baseline.json — the convention /gnosis:eval maintains. Flag anything that moved more than 2 points. If the stored baseline's keys don't match what the command prints, say so and report the current numbers alone; never diff mismatched shapes.

2. Use the real corpus when the question is about the user's docs

python tests/bench/bench_real_corpus.py \
  --corpus ./docs \
  --golden ./golden.jsonl \
  --modes keyword,hybrid \
  --chunk-size 2000 \
  --out bench-results/eval-<label>.json