rag-eval

/rag-eval

Run the retrieval regression gate against the current repo state and report whether a recent change helped, hurt, or held steady.

When to invoke

• User runs /rag-eval or /rag-eval <label>
• User has changed files under ragcore/, hitgate/, or retrieval config and is about to commit or push
• User asks "did this change affect retrieval quality?" or "is it safe to ship?"

Steps

1 — Determine the label

Use the argument if provided, otherwise use rolling.

2 — Run the gate

bash hitgate/check.sh <label>

Set env vars if configured for a non-default corpus or retriever (see README.md in this skill folder):

RAG_SOURCE_ROOTS="..." RAG_EVAL_DATASET="..." RAG_EVAL_BASELINE="..." EVAL_EXTRA_FLAGS="..." \
  bash hitgate/check.sh <label>

If the command exits non-zero AND no baseline file exists at the configured path, skip to the No baseline branch below.

3 — Read the structured verdict

cat hitgate/<label>.verdict.json

4 — Report in plain language

Pass (verdict: "pass"):

Gate passed. Hit@5 held [base → current]. MRR [base → current]. [Note any improvement in Hit@1 or MRR if improvements list is non-empty.]

Improvement (verdict: "improvement", refreeze_recommended: true):

Gate passed and Hit@5 improved [base → current, +Xpp]. The frozen baseline is now stale in the positive direction — consider re-freezing:

cp hitgate/<label>.json hitgate/baseline.example.json

Regression (verdict: "regression"):

Regression: [for each item in regressions, state scope + metric + delta in pp]. Next: run the eval in verbose mode to see which cases are now missing:

python -m hitgate.run --verbose --label <label>

Then inspect the MISS rows for the affected intent class.

No baseline found (baseline path does not exist):

No baseline at [path]. To create one:

python -m hitgate.run --label baseline-v1
cp hitgate/baseline-v1.json hitgate/baseline.example.json

Then re-run /rag-eval to compare against it.