skill-eval

skill-eval

Runs headless claude -p agents on a suite of prompts, once per arm, then scores each run.

Arm	What the agent sees
with	plugins/<plugin> from the working tree
without	no plugin
against	plugins/<plugin> at --against <git-ref>, for old-vs-new comparisons

Every run uses a fresh temp workdir outside the repo with --setting-sources project,local --strict-mcp-config. User plugins, hooks, MCP servers and the global CLAUDE.md are not loaded, so the only non-built-in skill an arm can see is the one under test. Trigger rates are therefore measured without competition from other installed skills.

Run

EVAL=.claude/skills/skill-eval/skill-eval.py
python3 $EVAL run evals/<plugin>/<suite>.json --model sonnet --reps 3 --budget 1
python3 $EVAL run evals/<plugin>/<suite>.json --arms with,against --against HEAD   # did my uncommitted edit help?
python3 $EVAL grade evals/<plugin>/runs/<run-id>                                   # re-grade after editing checks/rubric

run --help lists every flag. Output goes to evals/<plugin>/runs/<suite>-<timestamp>/ (gitignored): summary.md holds the table, and each <case>/<arm>-<rep>/ holds transcript.jsonl, run.json (tool calls, cost, turns, final reply), grading.json and work/ (files after the run).

Each run is a paid model call. Run 1 rep on one --case first, read its transcript, then scale up.

Suite format

evals/<plugin>/<suite>.json. Example: evals/aio-anti-slop/prose.json.

{
  "plugin": "aio-anti-slop", "skill": "aio-anti-slop",
  "permission_mode": "acceptEdits", "allowed_tools": ["Bash"],
  "cases": [{
    "id": "readme-robotic",
    "prompt": "Our README.md sounds robotic ...",
    "files": {"README.md": "seed content"},
    "trigger": true,
    "checks": [{"desc": "install command kept", "file": "README.md", "regex": "bun add tinyq"},
               {"desc": "answers O(log n)", "output": true, "not_regex": "..."}],
    "rubric": ["README.md adds no new factual claims"]
  }]
}