skill-evaluator

Skill Evaluator

Evaluate local skills with a repeatable loop: inspect structure, read eval definitions, run one or more realistic prompts, then score the output with explicit assertions and evidence.

When to Use

• Test whether a skill is wired correctly
• Check whether evals/evals.json exists and is usable
• Run a real prompt through a skill and grade the result
• Compare a skill against a no-skill baseline or older snapshot
• Identify missing folders, weak evals, and flaky assertions

Required Inputs

At minimum, identify:

SKILL_PATH: absolute or workspace-relative path to the skill directory
GOAL: structure check / eval review / live run / baseline comparison

Evaluation Workflow

1. Structure Check

Confirm the skill directory is sane before judging outputs.

Expected layout:

skill-name/
  SKILL.md
  evals/evals.json                   # required
  references/evaluating-skills.md    # required for evaluator
  scripts/                           # optional but useful

Flag these issues explicitly:

• missing SKILL.md
• nested duplicate directory like skill-name/skill-name/
• evals/ exists but evals/evals.json is missing or invalid JSON
• eval cases missing id, prompt, or expected_output

2. Eval Review

Read evals/evals.json if present and assess whether each case is realistic.

Good evals include:

• a real user prompt
• a short success definition
• optional input files
• assertions that are concrete and checkable

Weak evals include:

• vague prompts
• purely subjective assertions
• no evidence path for pass/fail

3. Live Run

Run at least one representative prompt from the eval set or create a focused ad hoc prompt.

For each live run:

• load the target skill
• read only the files the skill itself points to
• produce the answer or output
• grade against assertions with evidence

4. Baseline Comparison

Always rerun the same prompt without the skill (or against a snapshot of the older skill) to establish a baseline.