skill-validator

Skill Validator — Ralph Loop

A continuous-iteration eval loop for skills. Geoffrey Huntley-style: run the skill, evaluate, fix, re-run until pass criteria are met or the budget is spent. The loop fails closed, not open — if the budget exhausts before pass, the skill is reported as failing with a numbered list of unmet criteria.

When to use

• After authoring a new skill, before promoting it for general use
• After editing an existing skill's prompt or pre-flight checklist
• When a skill behaves correctly in chat but fails in scripted runs (or vice versa)
• When promoting a skill from one project to another
• When validating that a skill works against a specific target repo

When NOT to use

• The skill is purely conversational with no measurable output — there's nothing to score
• The target repo is unavailable or empty — there's nothing to run against
• The skill is already covered by deterministic unit tests — those are cheaper and stricter

Inputs

• SKILL_PATH — path to the skill being validated (e.g., .claude/skills/quality-review/SKILL.md)
• TARGET_REPO — path to the repo the skill will run against
• PASS_CRITERIA — file or inline list of required behaviors. Each criterion is a yes/no the loop can score. Format: <id>: <description> (one per line)
• BUDGET — max iterations (default 5) and max minutes (default 30). Hard cap; the loop fails if either is hit.

The Loop

iteration = 0
while iteration < BUDGET.max_iterations:
    iteration += 1

    # 1. Run skill on representative input from TARGET_REPO
    output = run_skill(SKILL_PATH, sample_input(TARGET_REPO))

    # 2. Score output against PASS_CRITERIA
    results = score(output, PASS_CRITERIA)
    if all(results.values()):
        return PASS(iteration)

    # 3. Identify which criteria failed and why
    failures = [(id, why) for id, ok in results.items() if not ok]

    # 4. Generate a targeted fix to SKILL.md
    patch = propose_fix(SKILL_PATH, failures)
    apply(patch)
    log(iteration, failures, patch)

return FAIL(iteration, last_failures)