Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/akshay5995/agent-skill-evalsnpx agentmods add skills/akshay5995/agent-skill-evals/agent-eval-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/akshay5995/agent-skill-evals/agent-eval-skills)<a href="https://agentmods.dev/skills/akshay5995/agent-skill-evals/agent-eval-skills"><img src="https://agentmods.dev/badge/skills/akshay5995/agent-skill-evals/agent-eval-skills/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/akshay5995/agent-skill-evals/agent-eval-skills"><img src="https://agentmods.dev/badge/skills/akshay5995/agent-skill-evals/agent-eval-skills.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00047 | $0.00994 |
| Opus 5 | $0.00023 | $0.00497 |
| Sonnet 5 | $0.00009 | $0.00199 |
| Haiku 4.5 | $0.00005 | $0.00099 |
Grade A, and why
agent-eval-skills scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 79 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Agent Skill Evals
Build a tight red → evidence → green loop. Promptfoo is the runtime; agent-skill-evals supplies its provider, assertions, Test Pack, and static checks. Batch related reads and spend agent turns only on evidence that can change the eval.
1. Contract
Read the target SKILL.md, every file it points to, and existing project tests. State the smallest observable product promise and choose one branch: behavior, routing, or role-play.
Complete when the promise names an observable outcome and the selected branch matches it.
2. Scaffold
Detect the repository's package manager and install agent-skill-evals and promptfoo as development dependencies when absent. Run installed binaries through that package manager:
pnpm exec agent-skill-evals init --skill <path> --adapter <codex|claude-code|pi>
Adapt the command to the detected manager, such as npx agent-skill-evals for npm. Treat CLI help and generated files as the interface; source and built bundles are outside the setup path.
Confirm the selected Codex, Claude Code, or Pi CLI is installed and authenticated before the runtime eval. Package installation supplies the eval integration, not the agent executable or its credentials. For Claude Code, require CLAUDE_CODE_OAUTH_TOKEN from claude setup-token or ANTHROPIC_API_KEY; the isolated runtime cannot reuse a macOS Keychain login.
Complete when the generated config imports the package's public entry points and promptfoo eval remains the runtime command.
3. Red
Add the smallest realistic fixture. When final output cannot prove the promise, add a deterministic verifier that fails before the agent runs. Verify stable structure and facts, not incidental prose: normalize formatting, accept grammatical variation, and reject missing, misplaced, or invented facts. Exact bytes are the contract only when the target skill promises exact bytes.
When the promise is unrestricted semantic equivalence and the result is present in final output, use a native Promptfoo model-graded assertion under promptfoo.assert rather than growing a synonym list.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 79 lines · 47 tokens per session scan A 1dc188f8acd9
agent-eval-skills is a skill published in the GitHub repository akshay5995/agent-skill-evals (2 stars, last pushed 1mo ago), licensed MIT. It adds 47 tokens to every session and 994 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
evaluate-skill
Measure a skill's reliability — run it k times for a pass@k score, design or interpret its eval, or compare it against the base agent. Use when the user wants to run, design, or interpret a skill's eval, or write an .eval.yaml spec.
claude-code-smoke
Use when asked to run the Claude Code smoke evaluation.
grill-skill
Build and harden a skill with evals — interview to design its eval tasks, then run, measure, and iterate. Use when the user wants to create or improve a skill's eval, or run the create → test → improve loop for a skill.
setup-skills-evals
Set up the skills-evals library in a repository — discover agent artifacts, interview the user about what to test, scaffold eval cases, and wire CI and local runners. Use when the user wants to set up skills-evals, test their agent skills, add evals for skills or custom agents, check why a skill isn't triggering, or…
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code.
test-driven-development
Drives development with tests using the red-green-refactor loop. Use when implementing any logic, fixing any bug, or changing any behavior. Use when you need to prove that code works, when a bug report arrives, or when you're about to modify existing functionality.