eval

Eval

gnosis-mcp eval in ~10 lines of wrapper logic. Reports retrieval quality against your golden queries, compares to last known numbers, and points at /gnosis:tune if anything looks regressed.

Different from /gnosis:tune: tune sweeps configurations looking for the best chunk size / embedder / rerank combo. Eval just reports current numbers. Run eval often (after each ingest, whenever the corpus changes). Run tune occasionally (after corpus shape change, new embedder, weekend of experimentation).

Usage

/gnosis:eval                    # full run — print numbers + interpret + compare
/gnosis:eval quick              # numbers only, no interpretation
/gnosis:eval save               # save current result as new baseline
/gnosis:eval diff               # compare current to last saved baseline

Mode: $ARGUMENTS

Step 1 — run the harness

gnosis-mcp eval --json

Parses to:

{
  "queries": 10,
  "hit_at_5": 1.000,
  "mrr": 0.950,
  "mean_precision_at_5": 0.668,
  "ndcg_at_10": 0.871
}

If the command errors (no [embeddings] extra, no golden file, empty DB), explain the exact cause and the fix. Don't proceed with empty numbers — fail loudly.

Step 2 — interpret

Report as a compact table:

Interpretation thresholds (rough heuristics, corpus-dependent):

• Hit@5 ≥ 0.85 → healthy. Users rarely need to read past position 5.
• Hit@5 0.70-0.85 → acceptable. Keyword-saturated corpus likely; little room to grow without changing the eval set.
• Hit@5 < 0.70 → probably a corpus or chunking problem. Run /gnosis:tune to sweep chunk sizes.
• MRR / Hit@5 ratio < 0.80 → relevant docs exist in top-5 but ranked too low. Try title-prepending or reranking (/gnosis:tune covers both experiments).
• nDCG@10 ≥ 0.85 and Hit@5 = 0.92 matches our published dev-docs baseline — you're in the same regime as gnosis-mcp's own corpus.