Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/unshdee/proofrag/proofragnpx skills add unshDee/proofrag --skill proofraggit clone --depth 1 https://github.com/unshDee/proofragWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00095 | $0.01882 |
| Opus 5 | $0.00048 | $0.00941 |
| Sonnet 5 | $0.00019 | $0.00376 |
| Haiku 4.5 | $0.00010 | $0.00188 |
Grade A, and why
proofrag scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 150 lines — stays where its author put it; the contents beside it link to each section on GitHub.
proofrag
Turn "did my change make the RAG better or worse?" into one reproducible command. You (the agent) wire the user's app to the kit; the kit does dataset generation, judging, and reporting.
When to use
- User changed a prompt, model, chunker, embedder, or retriever and wants to know if quality moved.
- User has docs/a knowledge base but no evaluation set.
- User wants a hallucination/groundedness number, or a CI gate on answer quality.
Install the engine
This skill drives the proofrag CLI. Make sure it's on PATH (install once), or run
it ad-hoc with uvx:
uv tool install "proofrag[anthropic]" # or: pipx install "proofrag[anthropic]"
# no install needed: uvx "proofrag[anthropic]" demo
Use [openai] instead of [anthropic] for an OpenAI-compatible/local backend.
Credentials: ANTHROPIC_API_KEY (default, cheap Haiku judge) or OPENAI_API_KEY
(OPENAI_BASE_URL for local/Ollama). No key? proofrag demo renders a sample scorecard.
If both keys exist, auto-detection chooses Anthropic; use PROOFRAG_PROVIDER=openai
to override. Proofrag does not auto-load .env, so source it first.
The loop
-
Inspect and generate from the user's corpus.
proofrag corpus ./docsUse
--include,--exclude, and--no-gitignorewhen the docs tree is noisy. PDF loading needs theproofrag[pdf]extra; HTML is supported by default.Then generate:
proofrag generate --corpus ./docs --out goldenset.jsonl --n 20Produces JSONL:
{id, question, gold_answer, gold_contexts[], difficulty, sources[]}with tierssingle_doc/multi_doc/unanswerable, pluscontext_metadatafor each gold context. Commit this file — it is versioned. -
Validate the golden set before committing it.
proofrag validate --goldenset goldenset.jsonl --corpus ./docs --out validation.jsonThis checks the JSONL contract, duplicate ids/questions, answerable cases without gold contexts, unanswerable cases that still cite context, source coverage, and a stable fingerprint. It exits non-zero on hard errors; add
--strictto fail on warnings too.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2d ago First seen · 150 lines · 95 tokens per session scan A fb0bcc9dedc8
proofrag is a skill published in the GitHub repository unshDee/proofrag (2 stars, last pushed 23d ago), licensed MIT. It adds 95 tokens to every session and 1,882 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
generate-rag-dataset
Generate a synthetic evaluation dataset from your RAG knowledge base. Creates diverse Q&A pairs with expected answers and relevant context, ready for LangWatch experiments and platform import. Use when you need test data for your RAG pipeline.
rag-eval
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG…
autorag-setup
Configure AutoRAG for first use or repair its single-agent model, approved document roots, retrieval indexes, datasource skills, and health checks without exposing credentials.
autorag
Use an already configured AutoRAG librarian agent to search, summarize, compare, and answer questions from local document collections. Use autorag-setup for configuration or indexing changes.
rag-evaluator
Evaluate RAG pipeline quality across faithfulness, relevance, and hallucination metrics. Use when user asks to test, benchmark, or improve a RAG system, or when RAG outputs look wrong.
ai-engineering-toolkit
6 production-ready AI engineering workflows: prompt evaluation (8-dimension scoring), context budget planning, RAG pipeline design, agent security audit (65-point checklist), eval harness building, and product sense coaching.