Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/lyonzin/knowledge-rag/rag-evaluate-qualitynpx skills add lyonzin/knowledge-rag --skill rag-evaluate-qualitygit clone --depth 1 https://github.com/lyonzin/knowledge-ragWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00078 | $0.02557 |
| Opus 5 | $0.00039 | $0.01278 |
| Sonnet 5 | $0.00016 | $0.00511 |
| Haiku 4.5 | $0.00008 | $0.00256 |
Grade A, and why
rag-evaluate-quality scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 213 lines — stays where its author put it; the contents beside it link to each section on GitHub.
rag-evaluate-quality — measure, do not guess
When to use this skill
Trigger this skill:
- Weekly cadence — set a recurring reminder (Monday morning, Friday afternoon)
- After significant reindex activity — new content added, models changed, presets swapped
- When answer quality feels off — the user reports "search is worse than it used to be"
- After a version upgrade —
pip install -U knowledge-ragbump, worth confirming no regression - Before proposing tuning changes — if you are about to suggest
hybrid_alpha=0.5ormin_score=0.3, measure first
Do NOT run:
- Every session (waste of cycles; retrieval quality is stable session-to-session)
- On brand-new empty corpora (nothing to evaluate)
What this skill commits to
The agent produces a numeric quality report, not a vibe check. Outputs:
- Index health — chunks, cache hit rate, embedding model, dimensions
- Retrieval metrics — MRR@5, Recall@5, Precision@5 across a small set of representative queries
- Interpretation — what the numbers mean, whether they moved vs last run, what to do about it
Numbers get logged so the trend is visible over time.
Steps
-
Snapshot index health:
get_index_stats()Capture:
documents_count,chunks_countcache_hit_rate(higher after warmup = healthy)embedding_model,embedding_dim
-
Prepare or reuse an evaluation set. knowledge-rag's
evaluate_retrievaltool needs test queries + expected documents (ground truth). Two options:- Reuse a canonical set — if the project already has
tests/evaluation-queries.jsonor similar, load it. - Build a quick set inline — 5-10 queries that a domain expert (or the user) knows the "correct" answer document for.
Format (per the API contract):
[ {"query": "authentication design", "expected_docs": ["docs/adr/0018-auth.md"]}, {"query": "retry policy", "expected_docs": ["docs/adr/0031-retries.md"]}, ... ] - Reuse a canonical set — if the project already has
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- yesterday First seen · 213 lines · 78 tokens per session scan A f538e3762fdc
rag-evaluate-quality is a skill published in the GitHub repository lyonzin/knowledge-rag (262 stars, last pushed 3d ago), licensed MIT. It adds 78 tokens to every session and 2,557 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
go-naming
Go naming conventions and best practices. Use this skill when working with Go code and need to name packages, files, directories, structs, interfaces, functions, variables, or constants. Provides comprehensive naming guidelines following Go community standards.
rag-perf
Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass + aiperf load test driven by a single YAML config. Not for accuracy / RAGAS scoring (use rag-eval) or for deploying / repairing services (use rag-blueprint).
rag-blueprint
NVIDIA RAG Blueprint — deploy, configure, troubleshoot, and manage. Handles any RAG action: deploy, install, start, enable, disable, toggle, change, configure, troubleshoot, debug, fix, shutdown, stop, or tear down any RAG feature or service (Agentic RAG, VLM, guardrails, query rewriting, models, search, ingestion…
rag-eval
Filesystem RAG benchmarks: corpus/, train.json, evaluaterag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
creating-tasks
Scope a new task axis for the matrix: the column-direction in the reference × task projection. The skill is a single-process dialogue grounded in Jobs-to-be-Done discipline; the output is a task spec at corpus.commons/{corpus}/tasks/{task-slug}.md, ready to feed creating-applications for assembly into a compiled…
creating-applications
The single entry point for creating a new application: a compiled distribution built from a corpus subset projected onto a task axis (with optional lenses + practitioner-role agents). Run this when a user says "create an app for X" or similar. The skill first ensures a task spec exists (calling creating-tasks…