setup-skills-evals

Setting up skills-evals

You are helping the user adopt skills-evals — a zero-dependency library that validates, trigger-tests, and regression-tests agent artifacts (skills, instructions, custom agents, Cursor rules, prompt files). Follow this workflow in order. Ask the interview questions one at a time and wait for answers.

Step 1 — Inventory what exists

npx skills-evals list

Show the user the output. If nothing is discovered, stop and help them create their first artifact (e.g. a SKILL.md) before setting up evals.

Step 2 — Interview the user

Ask these questions one at a time:

1. "Which of these do you want to write eval tests for?" — show the discovered artifacts. For each one they pick, also ask:
   • "What are 3–5 things users actually say that should trigger it?" (realistic phrasing, not the description restated)
   • "What nearby asks should it NOT handle, and which artifact owns those instead?"
2. "Is this repo on CI?"
   • GitHub Actions → offer the ahnafyy/[email protected] action or a plain npx skills-evals run step.
   • Other CI (CircleCI, GitLab, Jenkins…) → add npx skills-evals run to their existing pipeline config.
   • No CI → skip; local runner only.
3. "How do you want to run it locally?" — offer to hook into whatever they use:
   • npm/pnpm/yarn → add a script to package.json
   • Gradle → add an Exec task
   • Make → add a target
   • plain shell → create an evals.sh

Step 3 — Scaffold and fill the eval cases

npx skills-evals init

This creates evals/cases/<name>.json stubs. Fill each one from the interview answers, following these rules:

• ≥ 3 positive prompts. Use the user's real phrasing. Never copy the artifact's description — that games the eval. Use "top_k": 1 for the artifact's signature ask.
• ≥ 2 negative prompts, each with "owner": "<other-artifact>" when another artifact should win — that makes it a real pairwise routing test.
• For glob-routed artifacts (applyTo instructions, Cursor globs rules) use { "path": "src/example.ts" } entries instead of prompts.
• Optionally add behavioral evals[] (Anthropic skill-creator schema): prompt, expected_output, expectations[], and files[] pointing into evals/fixtures/. Mark fixture-less evals "trust_level": "provisional".