testing-agent-skills

Testing Agent Skills

Use this skill when a customization asset is treated as done because it reads well.

A SKILL.md is a behavioural contract executed by a model. Nothing about it is verified by frontmatter linting, a markdown check, or a careful read. Every claim in a skill description - "use when X", "always applies", "never invents a root cause" - is a testable assertion, and until someone tests it the asset is untested code shipped to every user of the collection.

This skill treats the collection as the system under test. testing-llm-features covers the same discipline pointed at a product feature; the difference is what fails. A product eval failure ships a wrong answer to a customer. A skill eval failure ships a wrong process to every future task.

When to Use

• a skill, prompt, agent, or instruction file was edited and the behavioural effect is unknown
• a new asset is being added to a collection that already has near neighbours
• two assets compete for the same request and the wrong one keeps winning
• a description is being rewritten for discoverability and nobody can tell whether it got better
• an asset claims "always applies" and the claim has never been checked past turn three
• a collection has grown past the point where a human can spot-check it before release
• before publishing a collection, as a release gate
• the request mentions skill evals, prompt regression, activation testing, A/B on a prompt, or "does this actually change anything"

Smallest Useful Suite

A full suite is 10 to 20 cases with baselines and blind scoring. Nobody starts there. Six cases in an afternoon catch most of what is actually broken, and they establish whether the criteria are scorable at all - which is the thing you cannot learn by planning.

Start with exactly these six, in this order:

1. One activation-positive, phrased the way a user would ask, with the asset never named.
2. One activation-negative, on the nearest adjacent request the asset should leave alone.
3. One collision, against the nearest neighbour, with the intended winner stated up front.
4. One contract, on whatever the asset promises to produce.
5. One safety, if the asset touches brevity, speed, or autonomy - this is where confirmation steps quietly disappear.
6. One persistence, if the asset claims to always apply.