skill-testing

Skill Testing

An untested skill is a skill that silently breaks. Build a test harness covering frontmatter, content, and activation, run it in CI, and you'll catch regressions long before users file bugs.

When to Use

• Setting up a new skills repository
• Adding skills to an existing repo without test coverage
• Debugging why a skill exists but never activates
• Reviewing PRs that add or modify skills

Three Layers of Testing

1. Static validation — frontmatter parses, required fields present
2. Content lint — length, structure, code-block sanity
3. Activation testing — does the skill fire on realistic queries?

Layer 1: Static Validation

import matter from "gray-matter";
import { readFileSync } from "fs";
import { globby } from "globby";

async function validateAll() {
  const files = await globby("skills/**/SKILL.md");
  const errors: string[] = [];

  for (const file of files) {
    const raw = readFileSync(file, "utf-8");
    let parsed;
    try {
      parsed = matter(raw);
    } catch (e) {
      errors.push(`${file}: invalid YAML`);
      continue;
    }
    const { data } = parsed;
    if (!data.name) errors.push(`${file}: missing name`);
    if (!data.description) errors.push(`${file}: missing description`);
    if (data.description && data.description.length < 50) {
      errors.push(`${file}: description too short (${data.description.length} chars)`);
    }
    // Name must match directory
    const dirName = file.split("/").slice(-2)[0];
    if (data.name && data.name !== dirName) {
      errors.push(`${file}: name '${data.name}' doesn't match directory '${dirName}'`);
    }
  }

  if (errors.length) {
    console.error(errors.join("\n"));
    process.exit(1);
  }
}
validateAll();

Run on every PR. Catches 80% of authoring mistakes.

Layer 2: Content Lint

Check the body for common quality issues:

function lintBody(body: string, file: string) {
  const issues = [];
  if (body.length > 15_000) issues.push("body too long (>500 lines equiv)");
  if (!body.includes("## When to Use") && !body.includes("## When To Use")) {
    issues.push("missing 'When to Use' section");
  }
  if (!body.includes("## Best Practices")) {
    issues.push("missing 'Best Practices' section");
  }
  if (!/```/.test(body)) issues.push("no code example");
  // detect stale model IDs
  if (/claude-3[.-]/.test(body)) issues.push("stale model ID (claude-3-*)");
  // detect TODO markers
  if (/TODO|FIXME|XXX/.test(body)) issues.push("contains TODO marker");
  return issues.map((i) => `${file}: ${i}`);
}