claude-ruby-grape-rails eval-review.instructions.md

Eval Framework Review Rules

Audience: Agents, Not Humans

Imperative-only.

Eval Definitions (lab/eval/evals/*.json)

• Each JSON defines dimensions: completeness, accuracy, conciseness, triggering, safety, clarity
• Each dimension has a weight (float, all weights should sum to ~1.0) and an array of checks
• Check types must match functions in matchers.py MATCHERS dict
• Skill-specific checks override the default_eval() fallback in scorer.py
• Eval definitions should not exceed 3KB

Trigger Corpora (lab/eval/triggers/*.json)

• Each JSON has: skill, should_trigger, should_not_trigger, hard_should_trigger, hard_should_not_trigger
• Minimum counts: 4 should_trigger, 4 should_not_trigger, 2 hard_should_trigger, 2 hard_should_not_trigger
• Hard prompts must have an axis field with >= 2 distinct values across hard_should_trigger (typically "confusable" and "multi_step")
• Prompts must not contain skill names (routing contamination)
• No duplicate prompts within a file (normalized comparison)

Context Budget (lab/eval/context_budget.py)

• Advisory checks for CLAUDE.md size and aggregate routing-prompt char budget (8,000-char hard ceiling on the skill listing per CC routing budget)
• Zero API cost — file reads and frontmatter scanning only
• Wired into --changed, --all, and --ci modes in run_eval.sh

Module Map

The canonical module list is ls lab/eval/*.py. Re-derive on every review pass; do not rely on a frozen enumeration here.

When reviewing a change under lab/eval/:

1. Read the module's docstring and import block — they declare its role and whether it touches an LLM provider.
2. Check whether make eval-ci-deterministic reaches it transitively (see lab/eval/run_eval.sh --ci body and the allowlist in lab/eval/tests/test_eval_ci_determinism.py::DETERMINISTIC_PATH_FILES). Modules on that path MUST stay LLM-free.
3. LLM-bearing modules (behavioral_scorer.py, epistemic_suite.py, trigger_scorer.py --semantic path) are intentionally OFF the deterministic path. Confirm any new transport import goes into one of those, not into the deterministic path.