init-plugin-eval

Init Plugin Eval

Role: implementation. Create the eval scaffold so a plugin's review agents and advisory skills can be graded — without shipping any of it inside the plugin.

Constraints

1. Evals never ship. The eval tree lives at the repo root under evals/<plugin>/, never under plugins/<plugin>/. Build/test tooling inside a plugin dir is a shipping bug.
2. Stub, don't fake. Scaffold empty fixtures/ and expected/ dirs and a README. Do not invent fixtures — an empty scaffold must grade as "no fixtures", a clean no-op, not an error.
3. Never overwrite. If evals/<plugin>/ exists, report and stop.

Step 1 — Parse

• NAME = $0 (the plugin name, matching its plugin.json name).
• DIR = --dir value, else evals/<NAME>.
• If DIR exists → report and stop.

Step 2 — Create the structure

mkdir -p "$DIR/fixtures" "$DIR/expected"
for d in fixtures expected; do : > "$DIR/$d/.gitkeep"; done

• fixtures/ — input artifacts an agent/skill runs against (sample agent files, plugin dirs, prose use-cases).
• expected/ — one <fixture-stem>.json per fixture declaring the expected grading (which agent/skill it targets and the pass condition).

Step 3 — Write the grading-contract README

$DIR/README.md documents the contract a grader enforces:

# Eval corpus — <NAME>

Fixtures (`fixtures/`) and their expected gradings (`expected/*.json`) for the
<NAME> plugin's review agents and advisory skills.

## Grading contract

Each `expected/<stem>.json` pairs with a `fixtures/<stem>.*` input and declares:

- `fixture` — the fixture file/dir this grades.
- `description` — what the fixture exercises.
- `applicableAgents` / `applicableSkills` — the unit(s) under test.
- For a **review agent**: `expectedStatus` (`pass|warn|fail|skip`),
  `issueCount` ({min,max}), and `severities` bounds.
- For an **advisory skill**: the expected `recommendation`/fields and the
  rule citations or gates the output must contain.

A grader pairs every `expected/*.json` with a fixture and checks the actual
output falls within the declared bounds. An **empty corpus grades as
"no fixtures"** — a clean no-op, never an error.