Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/edonadei/caliper/evaluate-skillnpx skills add edonadei/caliper --skill evaluate-skillgit clone --depth 1 https://github.com/edonadei/caliperWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/edonadei/caliper/evaluate-skill)<a href="https://agentmods.dev/skills/edonadei/caliper/evaluate-skill"><img src="https://agentmods.dev/badge/skills/edonadei/caliper/evaluate-skill.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00060 | $0.00888 |
| Opus 5 | $0.00030 | $0.00444 |
| Sonnet 5 | $0.00012 | $0.00178 |
| Haiku 4.5 | $0.00006 | $0.00089 |
Grade A, and why
evaluate-skill scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 59 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Evaluate Skill
Run a skill repeatedly to measure how reliably it works, and design the evals that measure it.
Prerequisites
The caliper CLI must be on PATH. This skill can be copied into an agent without the Caliper repo, so do not assume the CLI is packaged with it. Install if missing:
pipx install caliper-eval
The engine (backend + model) is not part of the spec — it is chosen at run time with --model (skill) and --judge-model (judge), independently, from claude-code, codex, pi, defaulting to claude-code. Every backend is a CLI agent that uses its own subscription/auth; there is no direct-API backend (for API billing, configure a CLI with an API key). Full per-backend detail and every command: REFERENCE.md.
Spec shape
An .eval.yaml names the skill and a list of tasks. Keep skill.path relative to the spec file (usually ./SKILL.md):
skill:
path: ./SKILL.md # relative to the spec file
tasks:
- name: What success looks like
prompt: <prompt sent to the skill under test>
expect: <natural-language pass/fail criterion>
assert: | # optional deterministic Python check
assert ...
The spec has no backend/model or judge: block; pick the engine when you run, e.g. caliper run <spec> --model codex --judge-model codex. The full format (setup/cleanup, external assert scripts, sandbox) is in REFERENCE.md.
Bundled references
references/evals/ holds complete real examples (Claude Code smoke, commit workflow, screenshot, summarization, TDD) — each folder self-contained with its fixture SKILL.md and .eval.yaml. references/simple.eval.yaml is one compact multi-task spec.
No eval yet?
If the skill has a SKILL.md but no .eval.yaml, suggest the grill-skill workflow — it interviews the user and generates a happy/edge/adversarial spec. Use evaluate-skill directly when a spec already exists and the user wants to run, validate, report, or extend it.
What ships with it
13 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- evaluate-skill.eval.yaml 7.0 KB
- REFERENCE.md 23 KB
- references/evals/claude-code-smoke/claude-code-smoke.eval.yaml 660 B
- references/evals/claude-code-smoke/SKILL.md 253 B
- references/evals/commit-simple/commit-simple.eval.yaml 4.1 KB
- references/evals/commit-simple/SKILL.md 1.4 KB
- references/evals/screenshot/screenshot.eval.yaml 1.0 KB
- references/evals/screenshot/SKILL.md 7.6 KB
- references/evals/summarize/SKILL.md 2.1 KB
- references/evals/summarize/summarize.eval.yaml 1.7 KB
- references/evals/tdd/SKILL.md 9.6 KB
- references/evals/tdd/tdd.eval.yaml 4.9 KB
- references/examples/simple.eval.yaml 865 B
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 59 lines · 60 tokens per session scan A 14911ba6094f
evaluate-skill is a skill published in the GitHub repository edonadei/caliper (50 stars, last pushed 4d ago), licensed MIT. It adds 60 tokens to every session and 888 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
demo-reviewer
Demo skill for the Skill Eval Harness example. Use it to review a proposed change and label the severity of each finding.
agent-eval-skills
Evaluate an existing agent skill with Promptfoo-native behavior, routing, or role-play tests. Use when a skill needs executable evidence that its product promise works. Do not use to author the domain skill itself.
setup-skills-evals
Set up the skills-evals library in a repository — discover agent artifacts, interview the user about what to test, scaffold eval cases, and wire CI and local runners. Use when the user wants to set up skills-evals, test their agent skills, add evals for skills or custom agents, check why a skill isn't triggering, or…
bugfix-workflow
Use when the user reports a concrete bug in an existing application and asks for a code fix. Reproduce the bug, write or run a verifier that fails, change the smallest amount of code that makes the verifier pass, re-run the verifier, and explain the change. Do not use for: feature requests, code review, dependency…
test-driven-development
Write a failing test before implementing any fix or feature. Use when fixing bugs, adding features, or refactoring code that lacks test coverage.
fixtureless-check
Use when validating repository-level checks that do not need a copied fixture. Do not use for agent behavior.