eval

Eval

gnosis-mcp eval in ~10 lines of wrapper logic. Reports retrieval quality, compares to last known numbers, and points at /gnosis:tune if anything looks regressed.

Know what you are measuring. gnosis-mcp eval does not score your corpus. It loads nine hardcoded sample documents (SAMPLE_DOCS + SAMPLE_GIT_HISTORY_DOCS from the repository's tests/eval/ fixtures) into a temporary database and ignores GNOSIS_MCP_DATABASE_URL, so it prints identical numbers no matter what you have indexed. It is a harness smoke test, not a measurement of your docs. To score your own corpus, run the real-corpus benchmark (Step 1b) — that path is corpus-specific and reports nDCG@10, which gnosis-mcp eval never emits.

Different from /gnosis:tune: tune sweeps configurations looking for the best chunk size / embedder / rerank combo. Eval just reports current numbers. Run eval often (after each ingest, whenever the corpus changes). Run tune occasionally (after corpus shape change, new embedder, weekend of experimentation).

Usage

/gnosis:eval                    # full run — print numbers + interpret + compare
/gnosis:eval quick              # numbers only, no interpretation
/gnosis:eval save               # save current result as new baseline
/gnosis:eval diff               # compare current to last saved baseline

Mode: $ARGUMENTS

Step 1 — run the harness

gnosis-mcp eval --json

Parses to:

{
  "cases": 10,
  "hit_rate_at_k": 1.0,
  "mrr": 1.0,
  "mean_precision_at_k": 0.75,
  "k": 5
}

Those five keys are the whole contract — there is no ndcg_at_10 in this command's output, and no queries / hit_at_5 / mean_precision_at_5 (the real names carry _at_k plus the separate k field).

If the command errors, the cause is almost always a missing repository checkout: eval imports its fixtures from tests/eval/, so a plain pip install (no tests/ directory) exits 1 with an install-from-source hint. It does not need the [embeddings] extra, a golden file, or a populated database — it builds its own fixture in a temp DB. Don't proceed with empty numbers — fail loudly.