A guided workflow for the Claude plugin eval command, which runs test cases with and without a plugin and scores the results.

It helps developers preflight a run, validate cases, estimate cost, run the suite, read its JSON results, create a starter suite, and prepare CI instructions.

It checks the setup and test files before making model calls that can cost money. It also helps interpret the difference between using the plugin and not using it.

plugin-eval — Skill for Claude Code by melodic-software

Install

Getting it into your agent

This one installs as part of its plugin. Adding the marketplace and installing the plugin brings it with everything else the plugin ships.

Claude Code

/plugin marketplace add melodic-software/claude-code-plugins

Claude Code

/plugin install evals

Made for: Claude Code.

Or install evals, the plugin that ships this one along with the rest of its 4 skills.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/melodic-software/claude-code-plugins/plugin-eval/github.svg)](https://agentmods.dev/skills/melodic-software/claude-code-plugins/plugin-eval)

Your own site

<a href="https://agentmods.dev/skills/melodic-software/claude-code-plugins/plugin-eval"><img src="https://agentmods.dev/badge/skills/melodic-software/claude-code-plugins/plugin-eval/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/melodic-software/claude-code-plugins/plugin-eval"><img src="https://agentmods.dev/badge/skills/melodic-software/claude-code-plugins/plugin-eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 202 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 9,475 The whole file, excluding the scripts and references it only reads on demand.

Security scan D 2 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00202	$0.09475
Opus 5.5	$0.00081	$0.03790
Sonnet 5.5	$0.00040	$0.01895
Haiku 4.5	$0.00020	$0.00948

Measured yesterday against content hash d8d485255ad9, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade D, and why

plugin-eval scanned grade D with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.

The scan reads SKILL.md. This mod also ships 9 executable files (scripts/calibrate-judge.py, scripts/calibrate-judge.test.sh, scripts/noise-report.py, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Asks for rootmediumPrivilege escalation

A mod that escalates privileges can change anything on the machine, not only the project.

Preflight removes nothing and blocks on neither of the last two fields. When `leftover_sandboxes` is above zero, print `chmod -R u+w <tmp>/claude-eval-*` then `rm -rf <tmp>/claude-eval-*` for the user to run, note that a

Recursive force deletehighDestructive command

rm -rf with a variable or a broad path is one typo away from removing the wrong tree.

Preflight removes nothing and blocks on neither of the last two fields. When `leftover_sandboxes` is above zero, print `chmod -R u+w <tmp>/claude-eval-*` then `rm -rf <tmp>/claude-eval-*` for the user to run, note that a

plugins/evals/skills/plugin-eval/SKILL.md · 500 lines

How it starts

The opening of the file, as written. The whole thing — 500 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Run a plugin eval suite

claude plugin eval runs each case with the plugin and without it, grades each run, and writes aggregate-result.json. This skill does what the CLI does not, all before money is spent: checks that this machine can run the suite, prices the run, routes a target that is not a plugin, and reads the delta. It never re-implements grading and never simulates a run or a score.

Actions

Action	What happens
`preflight <target>`	Print the report below and stop. No CLI invocation, no spend
`validate [<eval-dir>]`	Run the static case validator only. No model call
`run <target>` (default)	Preflight, validate, print the estimate, then invoke the CLI
`read <json>`	Read a written `aggregate-result.json` in the order that keeps a delta honest
`ci`	Answer from the CI section below
`init [<name>]`	Scaffold a suite: `claude plugin eval init --bare <name>` where there is no terminal

This page holds every rule the actions use; answer from it. Each upstream specific it states has a row under Sources. Background for a human reader: case-authoring.md, reading-results.md (JSON fields), ci.md (workflow file, exit codes, parser), and records.md (the observations behind each source row).

Preflight

Print every field, in order, before anything else (the last two: probe):

cli_version:         <claude --version>
floor_met:           <true | false>
git_version:         <git --version | none>
platform:            <windows | wsl2 | linux | darwin | other>
sandbox_backend:     <present | absent | not-ready | unknown> (missing: <binaries that did not resolve> | links: <paths>)
target_type:         <plugin | wrapped-skill | wrapped-agent | rules>
suite_tools:         <read-only | the gated tools the cases request>
same_model:          <yes | no | unknown | off> (tested <model>, judge <model>)
estimate_usd:        roughly <n> (cases x runs x arms, plus judge calls)
ceiling:             <n> USD | unlimited
leftover_sandboxes:  <n> sandboxes (<total size> in <tmp>) | unknown
tmp_free:            <free space on the temp dir's filesystem> | unknown

Read the full file on GitHub · 500 lines

Files

What ships with it

29 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

yesterday Changed · +1 lines scan A → D d8d485255ad9
3d ago Changed · +1 lines 7669103f6553
5d ago Changed · +9 lines 2746d53733aa
7d ago Changed · -11 lines · -35 tokens per session eb512c457128
8d ago Changed 6964aa63043a
10d ago Changed 022c10b2bfef
11d ago Changed f3477ae95236
14d ago Changed · +1 lines 5a5f465a4f93
16d ago Changed · +2 lines · +10 tokens per session 16326a27857b
17d ago First seen · 497 lines · 227 tokens per session scan A 6ab65c1d12ab

Subscribe to this mod's changes

plugin-eval is a skill published in the GitHub repository melodic-software/claude-code-plugins (26 stars, last pushed today), licensed MIT. It adds 202 tokens to every session and 9,475 once invoked, about $0.0008 per session on Opus 5.5. A static security scan graded it D with 2 findings (asks for root, recursive force delete). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-14.

Analytics and metrics Claude Code Testing agents ai-tools marketplace mcp plugins

benchmark-before-optimize

For performance work: measure before changing, profile to find bottlenecks, compare before and after.

jimmc414/claude-code-plugin-marketplace · 25 tokens

test-structural-invariants

For data structure validation: test lengths, relationships, constraints that must hold, verify setup is correct.

jimmc414/claude-code-plugin-marketplace · 27 tokens

test-with-examples

For example-driven development: test cases as specifications, input/output pairs, documentation through examples.

jimmc414/claude-code-plugin-marketplace · 23 tokens

verify-solution-properties

For solution checking: independent verification that result is correct, separate from how it was computed.