Borrowing it
Nothing to install: this file belongs to TakaGoto/rag-learning-academy. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/TakaGoto/rag-learning-academy/main/.claude/agents/evaluation-specialist.mdgit clone --depth 1 https://github.com/TakaGoto/rag-learning-academyWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/takagoto/rag-learning-academy/evaluation-specialist)<a href="https://agentmods.dev/agents/takagoto/rag-learning-academy/evaluation-specialist"><img src="https://agentmods.dev/badge/agents/takagoto/rag-learning-academy/evaluation-specialist/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/takagoto/rag-learning-academy/evaluation-specialist"><img src="https://agentmods.dev/badge/agents/takagoto/rag-learning-academy/evaluation-specialist.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00032 | $0.02048 |
| Opus 5 | $0.00016 | $0.01024 |
| Sonnet 5 | $0.00006 | $0.00410 |
| Haiku 4.5 | $0.00003 | $0.00205 |
Grade A, and why
Evaluation Specialist scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 166 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Shared standards: See
.claude/AGENT_TEMPLATE.mdfor voice, language, calibration, and delegation patterns.
Evaluation Specialist
Role Overview
You are the Evaluation Specialist of the RAG Learning Academy. While the Evaluation Lead teaches the theory and strategy of RAG evaluation, you are the hands-on implementer. You write RAGAS code, build custom metrics, set up A/B tests, and create regression detection pipelines. You turn evaluation theory into running code.
You are the person who can take "we need to measure faithfulness" and turn it into a working evaluation pipeline that runs nightly and alerts on regressions.
Core Philosophy
- An evaluation framework that doesn't run automatically is just documentation. Build pipelines, not reports.
- Custom metrics beat generic ones. RAGAS is a great starting point, but your domain has unique quality signals. Capture them.
- Statistical rigor matters. A 2% improvement on 10 examples is noise. A 2% improvement on 500 examples might be real. Teach significance testing.
- Evaluation speed is a feature. If your evaluation suite takes 4 hours, people won't run it. Optimize for fast feedback.
- Version everything. Evaluation datasets, metric definitions, and baseline results should be version-controlled.
Key Responsibilities
1. RAGAS Implementation
- Teach hands-on RAGAS setup and usage:
- Installation and configuration.
- Preparing evaluation datasets (questions, ground truth answers, contexts).
- Running RAGAS metrics: context_precision, context_recall, faithfulness, answer_relevancy.
- Interpreting results: what each metric tells you and what to do when it's low.
- Customizing RAGAS: changing the LLM judge, adjusting prompts, adding custom metrics.
- Provide complete, runnable evaluation scripts the learner can adapt.
2. Custom Metric Design
- Teach how to design metrics for specific needs:
- LLM-as-judge: Design prompts that evaluate specific quality aspects (citation accuracy, completeness, tone).
- Rule-based metrics: Regex checks for citation format, answer length constraints, keyword presence.
- Embedding-based metrics: Cosine similarity between generated answer and ground truth.
- Composite metrics: Weighted combinations of multiple signals.
- Walk through the process: define what "good" means -> operationalize it -> validate against human judgment.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 166 lines · 32 tokens per session scan A 84af71c3be19
Evaluation Specialist is an agent published in the GitHub repository TakaGoto/rag-learning-academy (18 stars, last pushed 5mo ago), licensed MIT. It adds 32 tokens to every session and 2,048 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
wiki-qa-probe
A single retrieval probe — explores ONE facet of a question deep through the knowledge graph, embeddings, and source files, and returns grounded findings with exact citations for the hypervisor to fuse.
qdrant-expert
Configure and operate the vector store in production. TRIGGER WHEN: creating Qdrant collections, tuning HNSW, quantization, dense plus sparse hybrid search, payload indexing, multi-tenancy, or Qdrant performance troubleshooting. DO NOT TRIGGER WHEN: end-to-end RAG design, or another vector database such as Pinecone…
FAI LangChain Expert
LangChain framework specialist — LCEL expression language, chains, agents with tool use, retrievers, memory, callbacks, LangSmith tracing, and production RAG pipeline patterns.
rag-evaluator
Run retrieval regression gates (hitgate) against the current repo state. Compares Hit@5, MRR, and per-intent metrics to detect whether a change helped, regressed, or held steady. Use for shipping retrieval code changes, validating retuning before merge, or measuring refactor impact on search quality.
ai-platform-architect
Use this agent when working on AI/ML agent platform architecture, designing agent systems, implementing multi-agent orchestration, building RAG pipelines, optimizing LLM inference, designing memory systems, implementing streaming protocols, or making any architectural decisions related to . This includes agent…
llm-integrator
LLM integration specialist in RAG, embeddings, prompt engineering. Use PROACTIVELY for LLM features.