create-skill-eval

create-skill-eval

Build the graded harness for one inner skill and freeze it into that skill's eval/ directory. This is step "instrument the baseline" factored out of the optimizer: the how to grade THIS skill lives here and ships with the skill; the how to run the loop lives in optimize-skill-loop and is skill-agnostic.

• You produce a conformant eval/ dir (see references/eval-contract.md).
• optimize-skill-loop consumes it read-only and never edits it.

The eval/ you freeze is the two halves of the loop's contract made concrete: it rides the skill's qvr version (so the loop can switch to a content version before each cohort), and its eval.py + rubric.yaml are what turn each captured qvr session → metrics ∈[0,1]. qvr is the loop's inner ledger — it owns the sessions, the cost, and the per-version identity; your harness only supplies the grading. You never plumb session ids or cost here; you author what correct means for this skill and hand it off frozen.

The eval is the inner skill's own asset. Grading text2sql (run the SQL, diff the rows), slugify (string match), and a codemod (build + test) have nothing in common — that difference belongs in the skill's eval/, authored here, not baked into the generic optimizer.

The deliverable: a conformant eval/ (the contract)

Everything you write must satisfy references/eval-contract.md — that's the seam the optimizer relies on. The frozen dir:

<skill>/eval/
  HARNESS.md          # manifest: metrics, axes, agents/N/models, exit, fixtures
  scenarios.jsonl     # frozen cases {id, input, expected?}
  eval.py             # deterministic grader CLI (--metric <id>, --explain)
  rubric.yaml         # frozen LLM-judge dimensions/weights/anchors
  <fixtures/builders> # whatever eval.py needs

Once frozen, the whole dir is byte-stable for the loop: git diff <baseline> <candidate> -- '*/eval/*' is empty forever after.

0. Prereq

Requires python3 on PATH — the grader you author (eval.py), its fixture builders, and scripts/validate-harness.py all use it (stdlib only, no pip install). PyYAML is optional: the validator falls back to a minimal manifest parser when it's absent.