minirag-pgvector-mcp: Skill for Claude Code

Measure retrieval quality against a golden set, and produce a markdown accuracy report per tool. This is the skill that keeps you from over-trusting the system.

When to invoke

• After any change to chunking, embedder, k, reranking, or tool logic.
• Before merging anything that touches ingest/retrieve/mcp.
• On a schedule / in CI against a fixture corpus.
• Whenever you make a new real query worth capturing as ground truth → add it to the golden set.

Inputs

• golden_set — a golden/*.yaml file of cases:

  - query: "what did we decide about 振り替え scope?"
    doctype: meeting-notes        # optional filter
    expected_sources:             # ground truth
      - 2026-07-12-team-sync.md
    # or expected_chunk_ids: [...]
  

• tools — which tool(s) to score (search_context, …). Defaults to all.

Procedure

1. Load the golden set.
2. For each case, call the actual tool (the MCP tool definition, not a private copy) with its query/doctype/k.
3. Score:
   • hit@k — did an expected source/chunk appear in top-k?
   • MRR — reciprocal rank of the first correct hit.
   • recall@k — fraction of expected items found.
4. Aggregate across cases per tool.
5. Write packages/eval/results/<date>-<tool>.md: per-case table (query, expected, returned, pass/fail, rank) + aggregate metrics.
6. Persist to eval_runs for trend history.

Two layers — keep them separate

• Retrieval eval (deterministic): the above. Fast, reproducible, no LLM. This is the primary gate.
• Answer eval (LLM-as-judge): given retrieved context, is the generated answer correct vs. a reference? Non-deterministic. Run separately, label clearly, never let it contaminate retrieval numbers.

Guardrails

• Test the real tool clients call, not a reimplementation — otherwise you're testing a fiction.
• Golden set is version-controlled data. Grow it; never delete a case to make numbers look better.
• A passing run is evidence, not proof — inspect a few per-case rows each time.
• When comparing changes (e.g. rerank on/off), change one variable and re-run; record the delta in the report.