skill-quality-auditor

Skill Quality Auditor

Navigation hub for evaluating, maintaining, and improving skill quality with 9-dimension framework scoring.

Prerequisites

This skill drives the pantheon-skill-auditor CLI and is distributed by it (pantheon-skill-auditor skill install). Every audit command below requires that binary on PATH. Confirm it is available:

pantheon-skill-auditor --version

If it is missing, build or install it first (see Quick Start). There is no self-contained fallback for these commands.

Quick Start

Build once, then audit:

Build once:

bun run build:skill-auditor

Run audits:

# Single skill
pantheon-skill-auditor evaluate <domain>/<skill-name> --json --store

# Batch with grade gate
pantheon-skill-auditor batch <skill1> <skill2> --fail-below B --store

When to Use

• Evaluate skills before merge or publication using 9-dimension scoring
• Generate remediation plans, detect duplication (>20% threshold), or enforce CI quality gates
• Validate eval scenario coverage and artifact conventions

When Not to Use

• Write the skill first — do not audit an unfinished draft
• Avoid using this as a substitute for peer review of logic or domain accuracy

Workflow

1. Run pantheon-skill-auditor evaluate <skill> --json --store
2. Check artifacts and eval coverage using deterministic criteria
3. Generate a remediation plan with T-shirt sizing and score delta estimates
4. Run the auditor again to verify improvement; if score is below target, check remediation-plan.md and focus on the lowest-scoring dimension

Mindset

• Use scores as directional signals, not absolute truth.
• Apply deterministic, reproducible checks over manual review.
• Use threshold-based evaluation rather than relative comparisons.
• Keep audit rules strict for safety and consistency; stay flexible elsewhere.

Anti-Patterns (Summary)

• NEVER skip baseline comparison in recurring audits — WHY: score regressions go undetected without a prior audit.json
• NEVER ignore Knowledge Delta below 15/20 — WHY: low D1 means the skill adds no value over LLM baseline
• NEVER apply subjective scoring — WHY: scores drift between evaluators and cannot be automated in CI
• NEVER create kitchen-sink skills covering unrelated tasks — WHY: broad scope kills D7 and prevents correct triggering
• NEVER use harness-specific paths in skill content — WHY: absolute paths break when installed in a different repo
• NEVER list references without "When to Use" conditions — WHY: unconditional loading bloats context and penalises D5