Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add aks-builds/quality-skills --skill llm-eval-testinggit clone --depth 1 https://github.com/aks-builds/quality-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/aks-builds/quality-skills/llm-eval-testing)<a href="https://agentmods.dev/skills/aks-builds/quality-skills/llm-eval-testing"><img src="https://agentmods.dev/badge/skills/aks-builds/quality-skills/llm-eval-testing/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/aks-builds/quality-skills/llm-eval-testing"><img src="https://agentmods.dev/badge/skills/aks-builds/quality-skills/llm-eval-testing.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00160 | $0.03067 |
| Opus 5 | $0.00080 | $0.01533 |
| Sonnet 5 | $0.00032 | $0.00613 |
| Haiku 4.5 | $0.00016 | $0.00307 |
Grade A, and why
llm-eval-testing scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 300 lines — stays where its author put it; the contents beside it link to each section on GitHub.
LLM Eval Testing
You are an expert in evaluating LLM-powered products — chatbots, RAG systems, agents, classifiers, summarizers. Your goal is to help engineers build useful, reproducible, grounded eval pipelines that catch regressions before they ship, without falling for the metric-theater that surrounds this space. Don't fabricate eval framework features, metric names, or model behaviors. When uncertain, point the reader to the framework's docs and current independent benchmarks.
Initial Assessment
Check .agents/qa-context.md (fallback: .claude/qa-context.md) before answering. Pay attention to:
- Product type — chatbot, RAG, agent, classifier, summarizer, structured-output. Eval strategies differ.
- Underlying model — Anthropic Claude, OpenAI GPT, Google Gemini, open-source (Llama, Mistral, Qwen), or multiple. Evals should be model-agnostic; the product behavior may be very model-specific.
- Failure modes — what's been wrong in production? Hallucination, off-topic responses, bad tool use, slow latency, cost spikes, safety incidents?
- Eval framework in use — none, LangSmith, LangFuse, DeepEval, Inspect AI, Braintrust, hand-rolled.
- Cost / latency budget — eval runs cost money (API calls + judge calls). Plan accordingly.
If the file does not exist, ask: product type, model(s), production failure modes seen, existing eval infrastructure, cost constraints.
What evals are (and aren't)
Evals = automated checks that compare an LLM-system's outputs to expected behavior on a curated dataset. They are the closest thing to "unit tests for LLM apps" but with important differences:
- LLM outputs are non-deterministic (even temperature 0 has variance across versions).
- Many failure modes are subjective ("is this answer helpful?") and don't have a single ground truth.
- Eval datasets need careful curation — the dataset is the spec.
- Judge models can be wrong; LLM-as-judge needs validation against human labels.
Evals are NOT:
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 300 lines · 160 tokens per session scan A 0ff35f0ae581
llm-eval-testing is a skill published in the GitHub repository aks-builds/quality-skills (2 stars, last pushed 5d ago), licensed MIT. It adds 160 tokens to every session and 3,067 once invoked, about $0.0008 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
test-case-writer
Use when someone asks to generate test cases, write test cases from a user story, create test cases from a BRD, design test cases from a mockup or wireframe, or produce a test case table from requirements.
automated-e2e-testing
A workflow for turning manual web-app test cases into Playwright end-to-end tests and running them. End-to-end tests check a complete user flow through the website.
test-strategy
A method for deciding how a feature should be tested by turning risks into testing scope, depth, and priorities. TDD, or test-driven development, is not the focus here; this works at the system level through UI, API, manual, and specialist testing.
api-testing
API testing checks a software service directly through its endpoints, using OpenAPI or Swagger documentation or automated test cases. It can produce and run scripts that test requests and responses.
regression-testing
A workflow for deciding which existing tests should run after a code change. Regression testing checks that a change has not broken features that already worked.
exploratory-testing
A method for exploratory testing, where a tester learns an unfamiliar system while looking for risks instead of following only predefined test cases. It produces structured notes about the system, risks, test ideas, bugs, and unanswered questions.