validate

Validate eval cases before a run spends money

A case that fails to load costs the same as one that works. scripts/validate-cases.py reads the suite off disk and reports what would break, with no model call and no network access.

Run it

python3 "${CLAUDE_PLUGIN_ROOT}/skills/validate/scripts/validate-cases.py" <eval-dir>

<eval-dir> is the directory holding the cases, evals/ under the plugin root by default. Add --json for the findings as an array of {level, case, file, message} objects; --help prints the full contract from the script's header.

Findings print one per line as <FAIL|WARN> <case>/<file>: <message>, sorted by case then file; a finding about the suite as a whole carries the eval dir in place of <case>/<file>.

What the two tiers mean

FAIL is what the binary itself rejects: an unknown prompt.md frontmatter key, a grader with no usable type, an unknown option for the declared grader type, a case with no grader, two graders sharing a name, a non-positive weight, runs / max_turns / timeout_seconds out of bounds, an env key that does not match EVAL_[A-Z0-9_]*, a case that has a case.yaml but no schema_version or name in it or in prompt.md frontmatter (without a case.yaml the binary supplies both), a case with no prompt (prompt.md body or execution.prompt), a grader missing an option its type requires, and a schema_version that is not a quoted version string or whose major is newer than the binary supports. The complete sets the script checks, so a question about a key or a type is answered here without opening the script:

• prompt.md frontmatter keys: schema_version, name, description, tags, plugins, runs, expected_outcome, model, max_turns, timeout_seconds, allowed_tools, append_system_prompt, env. A time limit is timeout_seconds; timeout is an unknown key, so the case fails to load.
• Grader type: regex, tool_order, tool_used, file_exists, llm, baseline. Any other value, such as contains or string_match, is no usable type, so the case fails to load. A substring or pattern check is type: regex with the text in pattern.
• Options per type, beside the type, weight, arm and name every grader takes: regex pattern, flags, match, target; tool_used tool, input_match, min, max; tool_order before, after; file_exists path, exists; llm criteria, focus; baseline baseline_file, criteria.
• Required options per type: regex pattern; tool_used tool; tool_order before, after; file_exists path; llm criteria; baseline baseline_file, criteria. A graders/*.md body stands in for pattern (regex) or criteria (llm, baseline).
• Bounds: runs 1 to 50, max_turns 1 to 200, timeout_seconds 1 to 3600.