run-evals

Run Evals

Run eval cases for a skill, grade each against assertions, and produce structured results in a deterministic directory layout.

Inputs

The user provides:

• Skill name — the slash-command name of the skill to test (e.g., plan-feature)
• Evals path — path to the evals.json file (e.g., evals/plan-feature/evals.json)
• Workspace — directory where results are written

Output Structure

Every run produces this exact layout — no variation:

<workspace>/
├── benchmark.json
├── feedback.json
├── summary.md
├── eval-1/
│   ├── grading.json
│   ├── timing.json
│   └── outputs/
│       └── (skill outputs)
├── eval-2/
│   └── ...
└── eval-N/
    └── ...

Process

Step 1 — Read evals.json

Read the evals file and extract:

• skill_name — the skill being evaluated
• evals[] — array of test cases, each with id, prompt, expected_output, files (optional), and assertions

Step 2 — Execute each eval case

For each eval in evals[], spawn a subagent with this prompt:

You are executing an eval for the /sdlc-workflow:<skill-name> skill.

Task: <eval.prompt>

<if eval.files>
Input files (read these before starting):
<for each file in eval.files>
- <evals_dir>/<file>
</for>
</if>

Write all outputs to: <workspace>/eval-<eval.id>/outputs/

Important:
- Invoke the /sdlc-workflow:<skill-name> skill via the Skill tool to process this task
- Write every output file to the outputs/ directory
- Do not interact with external services (Jira, Figma, etc.) — write to files instead

Parallelism: Spawn all eval subagents in a single turn so they run concurrently. Do not wait for one eval to complete before starting the next — the eval cases are independent.

When each subagent completes, capture total_tokens and duration_ms from the task completion notification immediately. Write to <workspace>/eval-<eval.id>/timing.json:

{
  "total_tokens": <value>,
  "duration_ms": <value>
}

Step 3 — Grade each eval case