add-eval

Add Eval

Add or update an LLM eval test for a specific tool or collection. This skill creates a new eval scenario or updates an existing one to cover new tools.

Use this skill when:

• You added a new tool with /add-tool and want eval coverage
• You modified a tool and need to update the eval scenario
• You want to add a new workflow scenario to an existing collection
• /build-evals skipped the collection because eval files already exist

Prerequisites

Before running, ensure:

1. The collection exists (src/umbraco-api/tools/{collection}/index.ts)
2. The project builds: npm run build
3. An LLM API key is available (eval tests use Claude Agent SDK — works with a Claude Code subscription or ANTHROPIC_API_KEY)

Arguments

• Collection name (required): the collection to add eval coverage for (e.g. /add-eval form)
• Tool or scenario hint (optional): what to test (e.g. /add-eval form "copy form workflow")

If no hint is provided, compare the tools in the collection against existing eval test coverage and suggest what's missing.

Agents

Agent	When to use
eval-test-creator	Creating or updating eval test files (Step 3 or 4)

Critical Rules

BUILD BEFORE RUNNING. Eval tests run against dist/index.js. Always npm run build first.

RUN COMMANDS SEPARATELY. Always run build and test as separate Bash calls. Never chain with &&.

ITERATE ON PROMPTS. Eval tests are probabilistic. If a test fails, the fix is usually in the prompt.

VERBOSE DURING DEVELOPMENT. Always set verbose: true when creating or debugging.

Workflow

Step 0: Check Eval Infrastructure

Check if the eval setup exists:

• tests/evals/helpers/e2e-setup.ts
• tests/evals/jest.config.ts

If the setup doesn't exist, tell the user to run /build-evals {collection} first to create the infrastructure. This skill does not create eval setup files.

Step 1: Understand Current Coverage

Read the collection's tools and existing eval tests: