Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
git clone --depth 1 https://github.com/kumaran-is/claude-code-onboardingWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/commands/kumaran-is/claude-code-onboarding/rag-eval-init)<a href="https://agentmods.dev/commands/kumaran-is/claude-code-onboarding/rag-eval-init"><img src="https://agentmods.dev/badge/commands/kumaran-is/claude-code-onboarding/rag-eval-init.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00019 | $0.03720 |
| Opus 5 | $0.00010 | $0.01860 |
| Sonnet 5 | $0.00004 | $0.00744 |
| Haiku 4.5 | $0.00002 | $0.00372 |
Grade A, and why
rag-eval-init scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 452 lines — stays where its author put it; the contents beside it link to each section on GitHub.
/rag-eval-init — Bootstrap RAG Evaluation
Scaffold a complete RAG evaluation setup. The goal is for the user to have something runnable within 30 minutes that catches regressions.
Phase 1: Confirm context
Ask:
- Where should
evals/live? (default: repo root) - Python or another language? (default: Python — matches FastAPI / ADK stack)
- Does a golden set already exist anywhere? (CSV, spreadsheet, Notion?)
- Vector DB / retrieval entry point? (so the runner can hit it)
Phase 2: Create the directory structure
evals/
golden_set.yaml # Hand-labeled queries
run_eval.py # Eval runner
metrics.py # Retrieval + answer metrics
judges.py # LLM-as-judge (v2, marked as not-yet-used)
baselines.json # Baseline scores for CI comparison
reports/ # Per-run reports (gitignored except .gitkeep)
README.md # How to add queries, run evals, interpret results
Phase 3: Write the files
evals/golden_set.yaml
Start with 10 placeholder queries spanning happy path / edge case / unanswerable. Show the schema clearly:
# RAG Golden Set
# Target: 50+ queries before tuning the system
# Coverage: ~60% happy path, ~20% edge cases, ~20% known unanswerable
version: 1
queries:
- id: q-001
question: "REPLACE WITH A TYPICAL USER QUERY"
category: happy_path
classification: factual_lookup
expected_documents:
- "REPLACE: document_id or path"
expected_sections:
- "REPLACE: section name"
expected_answer_contains:
- "REPLACE: phrase that must appear"
must_not_contain:
- "REPLACE: phrase that must NOT appear (hallucination check)"
expected_abstention: false
- id: q-002
question: "REPLACE WITH AN EDGE CASE — multi-hop or comparison"
category: edge_case
classification: comparison
expected_documents:
- doc-a
- doc-b
expected_answer_contains:
- "REPLACE"
expected_abstention: false
- id: q-003
question: "REPLACE WITH AN UNANSWERABLE QUERY"
category: unanswerable
expected_abstention: true
abstention_reason_should_mention: "insufficient evidence"
# ... add 7 more placeholders following the same pattern
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 452 lines · 19 tokens per session scan A b9bd294cc895
rag-eval-init is a command published in the GitHub repository kumaran-is/claude-code-onboarding (35 stars, last pushed 2mo ago), licensed MIT. It adds 19 tokens to every session and 3,720 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other commands, from other repositories
AI Evaluation Setup
Set up evaluation for an AI agent or application using Azure AI Evaluation SDK.
dark-factory-test-rag-response
Comprehensive test scenario 3 - verify RAG pipeline grounds answers in ingested video and surfaces citations.
proofrag
Evaluate a RAG/LLM app — generate a golden set, judge it, and produce a scorecard.
CLAUDE_CODE_F143_CHATBOT_ONLY
A task for fixing and strengthening an AI chatbot and FAQ system, adding regulation information retrieval, conversation history, and tests. RAG means retrieving relevant source content before generating an answer.
querying
Query documents from a search index using type-safe filters with support for pagination, sorting, field selection, scoring, and highlighting. Count matching documents efficiently without returning results.
agent
Add an AI agent / RAG backend (@convex-dev/agent) to the Convex app.