sw-eval

Specwright Eval Runner

Run eval cases by spawning subagents for skill invocation, then grading the results.

Usage

/sw-eval --suite skill                    # Run all skill evals
/sw-eval --suite skill --case sw-init-fresh-ts  # Run one case
/sw-eval --suite skill --dry-run          # List cases without running
/sw-eval --suite integration              # Run integration evals

Instructions

Parse the arguments from the command input. The arguments follow the patterns above.

Step 1: Load the eval suite

Run this to load and validate the eval suite (output goes to stderr):

python -m evals --suite <SUITE_NAME> --dry-run 2>&1

This prints each eval case ID and its fixture path. If --dry-run was requested, stop here and show the output.

If --case was specified, filter to only that case from the output.

Step 1b: Create results directory

Create a timestamped results directory:

python -c "
import os
from datetime import datetime, timezone
ts = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%S')
results_dir = os.path.join('evals', 'results', f'run-{ts}')
os.makedirs(results_dir, exist_ok=True)
print(results_dir)
"

Save this as RESULTS_DIR for all subsequent steps.

Step 2: For each eval case

For each eval case (from the dry-run output), do the following:

2a. Setup fixture

Copy the fixture to a temp working directory using Bash:

python -c "
import shutil, tempfile, os
fixture_path = os.path.join(os.path.dirname(os.path.abspath('evals/__init__.py')), 'evals', '<FIXTURE_PATH_FROM_DRY_RUN>')
workdir = tempfile.mkdtemp(prefix='eval-')
shutil.copytree(fixture_path, workdir, dirs_exist_ok=True)
print(workdir)
"

Save the workdir path for the next steps.

2b. Run the skill via subagent

Spawn a subagent using the Agent tool:

Agent(
  prompt="<THE EVAL PROMPT - resolve from evals.json prompt_template + prompt_args>

Work in this directory: <WORKDIR>
Change to this directory first, then perform the task.
Accept all defaults. Do not ask clarifying questions.",
  subagent_type="general-purpose",
  description="Eval: <EVAL_ID>"
)