Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add jaktestowac/awesome-copilot-for-testers --skill testing-llm-featuresgit clone --depth 1 https://github.com/jaktestowac/awesome-copilot-for-testersWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/jaktestowac/awesome-copilot-for-testers/testing-llm-features)<a href="https://agentmods.dev/skills/jaktestowac/awesome-copilot-for-testers/testing-llm-features"><img src="https://agentmods.dev/badge/skills/jaktestowac/awesome-copilot-for-testers/testing-llm-features/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/jaktestowac/awesome-copilot-for-testers/testing-llm-features"><img src="https://agentmods.dev/badge/skills/jaktestowac/awesome-copilot-for-testers/testing-llm-features.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector pass
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00108 | $0.02583 |
| Opus 5 | $0.00054 | $0.01291 |
| Sonnet 5 | $0.00022 | $0.00517 |
| Haiku 4.5 | $0.00011 | $0.00258 |
Grade A, and why
testing-llm-features scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 166 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Testing LLM Features
Use this skill when a product's behaviour depends on a model, and the current verification method is a person looking at output and deciding it seems fine.
The problem is not that LLM output is hard to assert on. It is that the same input produces different output, and a prompt or model change can degrade quality without breaking anything. No test in a normal suite fails. Coverage stays green. The feature just gets worse, and you find out from users.
An eval suite is the unit-test layer for this: a fixed set of cases, run on every change that could affect model behaviour, scored consistently enough that a regression is visible.
When to Use
- a feature calls a model and there is no automated check on output quality
- a prompt, model version, temperature, tool list, or retrieval config is about to change
- a model upgrade is proposed and nobody can say what would regress
- an agent or chain has grown beyond what manual spot-checks can cover
- RAG answers are drifting and it is unclear whether retrieval or generation is at fault
- a CI gate is needed for AI behaviour, not just AI code
Operating Principles
- Golden cases, not vibes. A case has an input, an expectation, and a scoring rule. "Looks good" is not an expectation.
- Determinism where you can get it. Pin the model version, set temperature to 0, fix seeds, freeze retrieval snapshots. Non-determinism is a property of the model; it is not an excuse for a non-reproducible test.
- Test the assertable layer first. Schema conformance, required fields, refusal behaviour, citation presence, tool-call shape, latency and cost. Most feature-breaking regressions are structural, and structural checks are cheap and exact.
- Rubrics before judges. If an LLM judges the output, it needs a written rubric with a scale, and the judge itself needs validating against human labels. An unrubriced judge is a random number with an API bill.
- A regression is a new failure on a case that used to pass. That is the gate. An absolute score threshold on a hard case set gates nothing useful.
- Separate retrieval failures from generation failures. A RAG answer can be wrong because the right document was never retrieved. Score retrieval independently or you will tune the prompt to fix an index problem.
- Cases are a product artifact. They encode what the feature promises. Review them like code and grow them from real failures.
- Cost and latency are results. A 12% quality gain for 4× the cost and 3× the latency is a product decision, so report all three.
What ships with it
3 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 9d ago First seen · 166 lines · 108 tokens per session scan A eeb4d4f09791
testing-llm-features is a skill published in the GitHub repository jaktestowac/awesome-copilot-for-testers (113 stars, last pushed 13d ago), licensed MIT. It adds 108 tokens to every session and 2,583 once invoked, about $0.0005 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
session-rag-eval
Run and debug Chatbox session attachment RAG model evaluation with synthetic and real long-file fixtures.
deepeval
Use when discussing or working with DeepEval (the python AI evaluation framework).
standalone-python-scripts
Skill "standalone-python-scripts" from iloveitaly/llm-ide-rules, covering standalone python scripts, /// script, requires-python = ">=3.13", dependencies = [] and ///.
fix-tests
Focus on all unit + command tests (pytest --exclude tests/integration). Make sure they pass and fix errors. If you run into anything very odd: stop, and let me know. Mutate test code first and let me know if you think you should update application code.
agentic-eval
Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when: Implementing self-critique and reflection loops Building evaluator-optimizer pipelines for quality-critical generation Creating test-driven code refinement workflows Designing rubric-based or LLM-as-judge evaluation systems…
agent-evaluation
Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks.