Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add BayramAnnakov/eval-coach --skill eval-coachgit clone --depth 1 https://github.com/BayramAnnakov/eval-coachWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/bayramannakov/eval-coach/eval-coach)<a href="https://agentmods.dev/skills/bayramannakov/eval-coach/eval-coach"><img src="https://agentmods.dev/badge/skills/bayramannakov/eval-coach/eval-coach.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00020 | $0.02843 |
| Opus 5 | $0.00010 | $0.01422 |
| Sonnet 5 | $0.00004 | $0.00569 |
| Haiku 4.5 | $0.00002 | $0.00284 |
Grade A, and why
eval-coach scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 344 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval Coach
An Agent Skill for designing comprehensive AI evaluation strategies using Evaluation-Driven Development (EDD).
Overview
Eval Coach guides you through a structured 5-step framework for evaluating LLM applications:
- Define Success - Map business goals to measurable metrics
- Design Dataset - Create diverse test cases (happy path, edge cases, adversarial)
- Select Methods - Choose Automated, LLM-as-Judge, or Human evaluation
- Plan Automation - Integrate evals into CI/CD
- Monitor Production - Track drift and collect feedback
When to Use This Skill
Invoke this skill when:
- Starting a new AI project and need an evaluation strategy
- Improving an existing agent's reliability
- Comparing different implementation approaches
- Setting up CI/CD for AI products
- Debugging production quality issues
Evaluation Philosophy
Capability vs Regression Evals
Two fundamentally different types of evaluations:
| Type | Starting Point | Goal | When Failure Occurs |
|---|---|---|---|
| Capability | Near 0% | Push higher over time | Expected - iterate and improve |
| Regression | Near 100% | Maintain - don't drop | Alarming - investigate immediately |
Key Insight: Convert production bugs into regression tests. Every bug you fix becomes a test that ensures it never happens again.
pass@k vs pass^k (Handling Non-Determinism)
Agents are probabilistic. Use the right metric:
pass@k = Probability of at least one success in k attempts
- Formula:
1 - (1-p)^k - As k increases, score rises - more "shots on goal" means higher odds of success
- Use for coding agents where pass@1 matters most (first try counts)
- Use when multiple solution attempts are acceptable
pass^k = Probability of succeeding every time in k attempts
- Formula:
p^k - As k increases, score falls - demanding consistency across more trials is harder
- Example: 75% per-trial rate × 3 trials = (0.75)³ ≈ 42% pass^3
- Use for customer-facing agents where users expect reliable behavior every time
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 344 lines · 20 tokens per session scan A d60152bd7830
eval-coach is a skill published in the GitHub repository BayramAnnakov/eval-coach (4 stars, last pushed 7mo ago), licensed MIT. It adds 20 tokens to every session and 2,843 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
deepeval
DeepEval evaluation workflow for AI agents and LLM applications. TRIGGER when the user wants to evaluate or improve an AI agent, tool-using workflow, multi-turn chatbot, RAG pipeline, or LLM app; add evals; generate datasets or goldens; use deepeval generate; use deepeval test run; send results to Confident AI…
deepeval-otel
Export raw OpenTelemetry traces from an AI application to Confident AI's Observatory. TRIGGER when the user wants to send OpenTelemetry or OTLP traces/spans from an LLM app, agent, RAG pipeline, or chatbot to Confident AI; configure the Confident AI OTLP endpoint; set confident.span. or confident.trace. attributes…
deepeval-tracing
Instrument an AI application with DeepEval's native tracing so its behavior is visible in Confident AI. TRIGGER when the user wants to add DeepEval tracing or @observe to an LLM app, agent, RAG pipeline, or chatbot; wire a framework, model-provider, or vector-database integration (LangGraph, LangChain, OpenAI Agents…
tracely
Instrument AI agents with Tracely and turn their production traces into CI gates. Use when the user mentions Tracely, tracely-ai, tracelysdk, the tracely CLI, or asks to trace/observe an AI agent, add LLM evaluators or LLM-as-a-judge columns, debug why a trace or conversation isn't showing up, wire agent regression…
LLM Tracing and Observability Setup
Configures end-to-end tracing for an LLM application using OpenTelemetry with LangSmith, Langfuse, or Helicone — span naming, metadata tagging, latency thresholds, and cost tracking.
llm-evaluation
LLM evaluation — automated metrics, human feedback, benchmarking. Use when testing performance, measuring AI quality, or establishing evaluation frameworks.