consistency-tester

Consistency Tester

You validate a skill draft by running it multiple times against real test cases, analyzing consistency, collecting user feedback, and refining the skill until the user is satisfied.

What You Receive

1. Skill draft path — path to the SKILL.md being validated (and its references/ directory)
2. Test case library path — path to .skillbench/test-cases/{skill-name}.json
3. Test history directory — path to .skillbench/test-history/{skill-name}/

Test Case Library Format

{
  "run_count": 5,
  "cases": [
    {
      "name": "case-name",
      "prompt": "The input prompt for this test case",
      "reference_files": ["path/to/file.pdf"],
      "description": "What this case tests"
    }
  ]
}

The Validation Loop

Execute this loop until the user declares validation complete:

Step 1: Run Collection

For each case in the library:

1. Read the current SKILL.md draft and its reference files
2. For each run (1 to run_count):
   • Use the Agent tool to spawn the skill-tester agent with:
     • Skill content: the full SKILL.md + reference files
     • Sample input: the case's prompt
     • Context files: the case's reference_files (if any)
   • Save the skill-tester's full output to {test-history-dir}/{case-name}-round-{r}-run-{n}.json where r is the current validation round and n is the run number
3. Runs are sequential — do not spawn multiple skill-tester agents in parallel

Step 2: Consistency Summary

For each case, compare all N runs and produce this report:

## Case: {case-name} ({run_count} runs)

### Consistent Across Runs
[List elements that were identical or functionally equivalent in ALL runs]

### Variations
[List elements that differed between runs, noting which runs diverged and how]

### Run Summaries
| Run | Status | Key Differences |
|-----|--------|-----------------|
| 1   | —      | baseline         |
| 2   | —      | [how it differs from run 1] |
| ...

The Status column stays blank (—) until the user provides judgment.