write-tests

Writing skillprobe Tests

Generate YAML test files for skillprobe. Always read the skill content first before writing tests.

When asked to write tests for a skill, generate BOTH files unless the user asks for only one:

1. A behavioral test file (for skillprobe run)
2. An activation test file (for skillprobe activation)

Behavioral Tests (for skillprobe run)

Test what the model DOES when the skill is loaded.

harness: claude-code    # or cursor
model: claude-haiku-4-5-20251001
timeout: 120
skill: ./path/to/skill  # directory with SKILL.md, or a .md file

scenarios:
  - name: "descriptive name"
    steps:
      - prompt: "what the user would type"
        assert:
          - type: contains
            value: "expected text"

Activation Tests (for skillprobe activation)

Test whether the skill LOADS for relevant prompts and stays quiet for irrelevant ones. This is a separate YAML format from behavioral tests.

harness: claude-code
model: claude-haiku-4-5-20251001
timeout: 60
skill: ./path/to/skill

activation:
  skill_name: my-skill
  should_activate:
    - "prompt that should trigger this skill"
    - "another relevant prompt"
  should_not_activate:
    - "completely unrelated prompt"
    - "another unrelated prompt"

The activation key replaces scenarios. These are two different file formats loaded by different commands (skillprobe run vs skillprobe activation). Never put both keys in the same file. skillprobe will either ignore one silently or fail to parse the file.

Bad:

skill: ./skills/commit
activation:
  skill_name: commit
  should_activate:
    - "commit my changes"
scenarios:
  - name: "commit happens"
    steps:
      - prompt: "commit"
        assert:
          - type: contains
            value: "commit"

Pick one format per file.

Testing skill combinations

Some tests need to load more than one skill at a time, either because you are probing interactions between skills or because the scenario legitimately needs both. Use a skills: list at the suite level instead of the singular skill::