plugin-evals

Plugin Evals

Measure what a plugin reliably does. A case is a realistic prompt plus one or more graders — pass/fail checks on the final reply, the transcript, tool calls, or created files. Runs are repeated and averaged, and every case also runs without the plugin so the scores show the plugin's contribution, not the model's baseline ability.

Suite layout

The suite lives under evals/ in the plugin root. One directory per case; nest cases under a plain directory to group them.

<plugin>/
├── plugin.json
├── skills/...
└── evals/
    ├── <case>/
    │   ├── prompt.md          # the prompt sent to the agent, plus run limits
    │   ├── case.yaml          # optional: workspace fixtures, history, extra dirs
    │   ├── graders/
    │   │   ├── criteria.md    # one grader per file
    │   │   └── skill-fired.md
    │   └── <fixtures, scripts referenced by case.yaml>
    ├── mocks/                 # optional: scripted MCP tool answers
    └── results/               # written by each run; add to .gitignore

Only prompt.md or case.yaml is required for a case to exist. Cases start from an empty working directory, so put everything the task needs in the prompt, in case.yaml fixtures, or in a fixture directory added via context.add_dirs.

Step 1 — Design the cases

Interview the user with ask_user_question:

1. Which skill, command, agent or MCP server the suite tests.
2. What a good result looks like, concretely — the artifact, its shape, the tool that must have run.
3. What should NOT trigger the plugin (negative cases: a near-miss request the skill should leave alone).
4. Whether the task needs workspace fixtures (a git repo, sample files) or an early conversation to continue.

Write the prompt the way a user would type it, phrased naturally — not naming the skill. A prompt that names the skill measures obedience, not triggering.

prompt.md frontmatter fields:

Field	Default	Purpose
name	directory name	Case name; report keys and --case filters use it
description	—	For humans; unused at run time
tags	[]	Filter with --tag
runs	3	Runs per arm, 1–50
model	default	The agent under test's model
max_turns	10	Turn cap; hitting it is recorded as a run error
timeout_seconds	300	Wall-clock cap per run
allowed_tools	[]	Tools the case may use, e.g. [Read, Glob, Grep, Skill]
env	{}	Extra environment variables, keys must match EVAL_[A-Z0-9_]*