Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/practicalswan/agent-skills/rag-evalnpx skills add PracticalSwan/agent-skills --skill rag-evalgit clone --depth 1 https://github.com/PracticalSwan/agent-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/practicalswan/agent-skills/rag-eval)<a href="https://agentmods.dev/skills/practicalswan/agent-skills/rag-eval"><img src="https://agentmods.dev/badge/skills/practicalswan/agent-skills/rag-eval.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00030 | $0.02612 |
| Opus 5 | $0.00015 | $0.01306 |
| Sonnet 5 | $0.00006 | $0.00522 |
| Haiku 4.5 | $0.00003 | $0.00261 |
Grade A, and why
rag-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 155 lines — stays where its author put it; the contents beside it link to each section on GitHub.
On-disk RAG evaluation (corpus/ + train.json)
Purpose
Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).
For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.
When not to use
Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).
Prerequisites
- Repo cloned; run commands from repo root (imports and paths assume this).
- Python 3.11+ and uv; eval deps:
uv sync --project scripts/eval. - Reachable RAG server and ingestor (defaults often
localhost:8081/8082). NVIDIA_API_KEYfor RAGAS (see credential hygiene); optionalRAG_EVAL_JUDGE_MODEL.- Dataset roots passed to
--dataset-pathseach containcorpus/andtrain.json.
Instructions
- Prepare data — Ensure each dataset directory matches the layout and
train.jsonrules inreferences/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents undercorpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there. - Run eval —
uv run --project scripts/eval python scripts/eval/evaluate_rag.pywith--dataset-paths,--host, and--port. Seereferences/benchmark-execution.mdfor command examples, outputs, and errors. Usereferences/evaluate-rag-cli.mdfor flag-level detail. - Tune quality — Adjust
--top_k/--vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature,--top-p,--max-tokens) as documented inreferences/benchmark-execution.mdwhen comparing retrieval/generation configs for RAGAS scores. - Analyze results — Use
references/result-analysis.mdfor scripts; scanrag_*_evaluation_summary.jsonfor headline RAGAS metrics. - Triage errors — Use the error signal table and the Troubleshooting section below.
What ships with it
10 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 155 lines · 30 tokens per session scan A 6fc714dace5f
rag-eval is a skill published in the GitHub repository PracticalSwan/agent-skills (13 stars, last pushed 4d ago), licensed MIT. It adds 30 tokens to every session and 2,612 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other skills, from other repositories
rag-evaluator
Evaluate RAG pipeline quality across faithfulness, relevance, and hallucination metrics. Use when user asks to test, benchmark, or improve a RAG system, or when RAG outputs look wrong.
rag-evaluation
Use this skill to evaluate the quality of a RAG pipeline on faithfulness, answer relevancy, context precision, context recall, and hallucination rate. Activates after a RAG system is implemented or when retrieval quality is in question. Produces a structured evaluation report with measurable results.
nemotron-retrieval-recipes
Use when planning, debugging, tuning, evaluating, exporting, or deploying public Nemotron embed/rerank retrieval recipes.
rag-evaluation
Evaluate retrieval-augmented-generation pipelines with reference-free metrics to detect silent retrieval and grounding failures.
skill-evaluator
评估和提升其他 Skill 的能力，提供基准测试、红队测试和自主改进循环（Karpathy Loop）.
generate-rag-dataset
Generate a synthetic evaluation dataset from your RAG knowledge base. Creates diverse Q&A pairs with expected answers and relevant context, ready for LangWatch experiments and platform import. Use when you need test data for your RAG pipeline.