eval

Eval — Run skill evals and score them

Turns the dormant modules/<slug>/evals/evals.json files into a real, graded scorecard: does the skill trigger on its prompts, and does its output satisfy each assertion?

The deterministic engine is scripts/eval-harness.py (discovery, validation, work-list, scorecard rendering — no model calls). Grading is your job: fan the cases out to subagents and have Haiku 4.5 judge every assertion with structured output. Installed path is ~/100xprism/scripts/eval-harness.py; in a checkout it's scripts/eval-harness.py.

Phase 0 — Pick the target

# one module, everything, or just what changed on this branch:
python3 ~/100xprism/scripts/eval-harness.py validate --module <slug>
python3 ~/100xprism/scripts/eval-harness.py validate --all
python3 ~/100xprism/scripts/eval-harness.py validate --changed origin/main

Fix any structural errors before grading — a malformed eval file can't be scored.

Phase 1 — Get the work-list

python3 ~/100xprism/scripts/eval-harness.py plan --module <slug> --json

This emits { "modules": [ { "module", "cases": [ { id, prompt, expected_output, assertions[], files[] } ] } ] }. Each (case, assertion) is one unit of work.

Phase 2 — Grade with parallel subagents (Haiku 4.5)

For each case, dispatch one subagent (use the subagents skill / Agent tool, or a Workflow fan-out) that:

1. Runs the prompt against the skill. Load the target skill (its SKILL.md) as context, then answer the case prompt exactly as the assistant would — this is the candidate response. Note whether the skill would have auto-triggered on that prompt (trigger accuracy) separately from output quality.

2. Grades each assertion. Spawn a Haiku 4.5 grader (model: claude-haiku-4-5) that, given the prompt, the candidate response, the expected_output, and one assertion, returns structured output:

   { "module": "<slug>", "case_id": <id>, "assertion": "<text>", "passed": true|false, "reason": "<one line>" }