What SkillSpector found — and what happened when we checked it

So we checked its flags — 96% were false

We drew a 240-skill sample spread across every category it flags — weighted to the most-starred skills, the ones most likely to be genuinely fine — and had a second model (Claude Haiku 4.5) read each skill in full against SkillSpector’s findings, then read the borderline verdicts ourselves. About 96% of its high-severity flags were false positives, and every single category came out above 85% false.

The reason is not a bad scanner — it’s the wrong surface. A SKILL.md is mostly prose, and a static scanner matches words, not meaning. It cannot tell a skill that runs curl … | bash from one that documents a vendor’s official installer; a defensive rule that says “never send secrets” trips the exfiltration pattern; a markdown table trips “hidden instructions”; a skill whose subject is agents trips “rogue agent”. The genuine risk in a skill usually lives in the scripts it ships, not the prose — and those we did not scan here.

So SkillSpector’s flag no longer raises a red mark on a skill on its own. It is shown as a caution to read, next to our own grade and, where present, the AI review — and the full findings, with the line each matched, are on every skill’s page so you can judge them yourself. A red flag here comes from a human-reviewed confirmation, not a pattern match.

What it flags most — and how often that was false

Across the skills it rated high or critical, by category. The last column is the false-positive rate for that category in our 240-skill check.

Rated high or critical — 12,650, top 100 by stars

SkillSpector’s own top-severity calls — shown here as a caution, not a verdict. Of these, 12,145 are skills our own scan graded A or B; in our sample, flags like these were false about 96% of the time.

How this was made

Each skill’s SKILL.md was scanned by SkillSpector in static mode (--no-llm) on our own hardware — no data left the machine, no rate limit. To measure its accuracy we drew a 240-skill sample stratified across every finding category (weighted toward the most-starred skills) and had Claude Haiku 4.5 read each skill in full against the findings, deciding real versus false positive; we then read every borderline verdict ourselves. ~96% were false, every category above 85%, so a high or critical finding is capped at a warn pill (a caution) rather than a fail. The raw severity and the full findings — the line each matched and how to fix it — are kept on every skill’s page under “Third-party audits”. SkillSpector is Apache-2.0; this is our own run of it, and the accuracy check is ours too.

How the audits work · All skills