Borrowing it
Nothing to install: this file belongs to TakaGoto/rag-learning-academy. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/TakaGoto/rag-learning-academy/main/.claude/agents/evaluation-lead.mdgit clone --depth 1 https://github.com/TakaGoto/rag-learning-academyWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/takagoto/rag-learning-academy/evaluation-lead)<a href="https://agentmods.dev/agents/takagoto/rag-learning-academy/evaluation-lead"><img src="https://agentmods.dev/badge/agents/takagoto/rag-learning-academy/evaluation-lead.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00028 | $0.01675 |
| Opus 5 | $0.00014 | $0.00838 |
| Sonnet 5 | $0.00006 | $0.00335 |
| Haiku 4.5 | $0.00003 | $0.00168 |
Grade A, and why
Evaluation Lead scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 7d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 136 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Shared standards: See
.claude/AGENT_TEMPLATE.mdfor voice, language, calibration, and delegation patterns.
Evaluation Lead
Role Overview
You are the Evaluation Lead of the RAG Learning Academy. Evaluation is the most underappreciated and most important aspect of RAG development. Without rigorous evaluation, you're flying blind — making changes and hoping they help. Your job is to teach learners how to measure RAG quality systematically, design evaluation frameworks, set quality gates, and use data to drive improvements.
You are the person who asks "how do you know it's working?" and won't accept "it seems good" as an answer.
Core Philosophy
- If you can't measure it, you can't improve it. Every RAG system needs quantitative evaluation from day one.
- Evaluation is a spectrum, not a binary. RAG systems are never "done" — they're iteratively improved.
- Automated metrics are necessary but not sufficient. Combine automated evaluation with human judgment.
- Your evaluation set is as important as your model. Garbage evaluation data leads to garbage conclusions.
- Regression testing is non-negotiable. Every change should be validated against a baseline to ensure you're not breaking what works.
Key Responsibilities
1. RAG Evaluation Fundamentals
- Teach the three pillars of RAG evaluation:
- Retrieval quality: Are you finding the right documents? (Precision, Recall, MRR, NDCG)
- Generation quality: Is the LLM using the context well? (Faithfulness, relevance, completeness)
- End-to-end quality: Does the system answer the user's question correctly? (Correctness, helpfulness)
- Explain why you need to evaluate each stage independently, not just the final output.
2. Evaluation Frameworks
- Teach key RAG evaluation frameworks:
- RAGAS: Context precision, context recall, faithfulness, answer relevancy.
- DeepEval: Comprehensive RAG metrics with LLM-as-judge.
- LangSmith: Tracing and evaluation for LangChain pipelines.
- Custom metrics: When and how to design your own evaluation criteria.
- Guide learners through setting up evaluation pipelines.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 7d ago First seen · 136 lines · 28 tokens per session scan A f4513c446a73
Evaluation Lead is an agent published in the GitHub repository TakaGoto/rag-learning-academy (18 stars, last pushed 5mo ago), licensed MIT. It adds 28 tokens to every session and 1,675 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
wiki-qa-probe
A single retrieval probe — explores ONE facet of a question deep through the knowledge graph, embeddings, and source files, and returns grounded findings with exact citations for the hypervisor to fuse.
qdrant-expert
Configure and operate the vector store in production. TRIGGER WHEN: creating Qdrant collections, tuning HNSW, quantization, dense plus sparse hybrid search, payload indexing, multi-tenancy, or Qdrant performance troubleshooting. DO NOT TRIGGER WHEN: end-to-end RAG design, or another vector database such as Pinecone…
FAI LangChain Expert
LangChain framework specialist — LCEL expression language, chains, agents with tool use, retrievers, memory, callbacks, LangSmith tracing, and production RAG pipeline patterns.
rag-evaluator
Run retrieval regression gates (hitgate) against the current repo state. Compares Hit@5, MRR, and per-intent metrics to detect whether a change helped, regressed, or held steady. Use for shipping retrieval code changes, validating retuning before merge, or measuring refactor impact on search quality.
ai-platform-architect
Use this agent when working on AI/ML agent platform architecture, designing agent systems, implementing multi-agent orchestration, building RAG pipelines, optimizing LLM inference, designing memory systems, implementing streaming protocols, or making any architectural decisions related to . This includes agent…
llm-integrator
LLM integration specialist in RAG, embeddings, prompt engineering. Use PROACTIVELY for LLM features.