skill-check

SkillCheck — Comprehensive Skill Testing

SkillCheck audits an entire skill directory, understands every moving part, and generates targeted tests to verify the skill works as advertised. It doesn't just check "does it roughly work" — it tests every aspect: triggering, input handling, output quality, script correctness, security, best practices, and resource integrity.

---

How it works

Three phases: Discover → Test → Report. Discovery is always required. Testing depth depends on what the user wants. Reporting adapts to what was found.

Depth	What it covers
Quick	Static analysis — structure, syntax, integrity, security, best practices
Standard	Static + script execution + happy-path I/O + eval regression
Deep	Everything — edge cases, mock files, e2e workflows, stress

If the user doesn't specify, default to Standard — it catches real functional problems without the time investment of Deep. If they seem in a hurry, go Quick. If they mention thoroughness, edge cases, or "test everything", go Deep.

For a worked example of a Standard Check from start to finish, see references/example-run.md.

---

Gotchas

• The evals.json format must match the skill-creator schema exactly (see references/eval-schema.md). Don't invent a new format — compatibility with skill-creator's benchmarking tools is the whole point.
• Mock file generation for binary formats (DOCX, XLSX, PPTX, PDF, images) requires third-party Python packages that may not be installed. Always check availability before generating and fall back to text-based equivalents if installation fails. See references/mock-files.md for the dependency list.
• Cross-reference mismatches between SKILL.md and scripts are often the root cause of multiple downstream test failures (script execution fails, eval regression fails, I/O tests fail — all for the same reason). Identify and report the root cause rather than listing each symptom as a separate unrelated failure.
• Best practice scores are advisory and reported separately — do not add them to the main pass/fail verdict percentage. A skill can be functionally perfect but structurally messy, and the report should reflect that distinction.
• Security review runs before script execution for a reason — if Critical security issues are found in scripts, warn the user before executing them at Standard or Deep depth. Don't silently run code you've just flagged as dangerous.
• Incomplete skills (TODOs, stubs, placeholder text) need a progress report, not a wall of failures. It's easy to mechanically run the standard flow and produce 15 failures that all say "this doesn't exist yet" — that's not useful. Read references/incomplete-skills.md and reframe.
• On Claude.ai without subagents, end-to-end tests run sequentially in shared context. This means earlier test results can influence later tests — an agent that just discovered a cross-reference bug might "know" to use the correct arguments on the next test. Keep this in mind when interpreting e2e results.
• The frontmatter name field must match the parent directory name per the Agent Skills spec. This is easy to miss because many platforms will still load the skill with a mismatch — but others won't, making it a silent portability bug.
• Report sections should be collapsed when empty — a Quick Check on a minimal skill should produce 5 sections, not a 12-section skeleton full of "N/A". See references/report-format.md for collapsing rules.