skill-quality-scorer

Skill Quality Scorer

Goal

Score a skill on quality, not domain sophistication. Review whether another agent could trigger and use the skill reliably with minimal ambiguity and minimal context waste.

Quick Workflow

1. Read the target SKILL.md.
2. If the skill documents a real callable or CLI, inspect the live interface before trusting notebook-derived claims.
3. For data workflows, try to follow the skill on representative data before final scoring.
4. Read only the referenced files needed to judge execution quality, resource partitioning, validation quality, source grounding, and empirical executability.
5. Score each required dimension using references/rubric.md.
6. Apply hard gates before computing the final verdict.
7. Return a compact scorecard plus the highest-signal revision guidance.
8. For every dimension score, include an explicit rationale directly under that score in the report.
9. Write a human-readable Markdown score report to the current working directory instead of hiding it inside the generated skill folder.

Scope

Use this skill to review:

• a skill folder
• a standalone SKILL.md
• a proposed skill diff
• two competing skill designs
• a notebook-derived skill after conversion

Do not score:

• the scientific correctness of the underlying domain unless the skill itself makes unsupported domain claims
• code quality outside the skill's own bundled scripts and instructions
• repository-wide documentation unrelated to the skill

Review Rules

• Default to reading SKILL.md first.
• Read references/ only when the main skill explicitly depends on them or when resource partitioning is being scored.
• Read scripts/ only when the skill relies on them for core execution or validation.
• When the skill claims concrete function or CLI behavior, verify that behavior against source, help(...), or -h/--help before treating the skill as complete.
• For notebook-derived data workflows, do not stop at interface inspection if representative data can be constructed or loaded reasonably.
• Treat missing branch coverage for method, backend, mode, provider, or similar parameters as a real usability risk, not a minor omission.
• Do not reward verbosity.
• Do not reward domain difficulty.
• Do not assume missing validation or compatibility notes are present elsewhere.
• Penalize trigger ambiguity, execution ambiguity, and hidden dependencies aggressively.