skill-creator

Skill Creator

A skill for creating new skills and iteratively improving them.

At a high level, the process of creating a skill goes like this:

• Decide what you want the skill to do and roughly how it should do it
• Classify the change into the lowest verification tier that can falsify its likely failure modes
• Write a draft of the skill
• Validate with the smallest evidence that can falsify the changed behavior; treat the full paired eval pipeline as separately authorized work, not an automatic consequence of a tier label
• Help the user evaluate qualitative or quantitative results when the selected tier produces them
• Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)
• Repeat until you're satisfied
• Escalate the verification tier only when the current evidence cannot resolve the changed behavior

Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through the applicable stages. Full A/B benchmarking is a capability, not a tax on every edit. Words such as "optimize", "improve", or "comprehensive" describe intent, not failure surface or evaluation budget. Do not create eval files, fan out paired agents, grade outputs, or launch a viewer merely because a task sounds broad or a long conversation precedes it.

Six standing disciplines apply throughout, because these failure modes ship convincing-looking skills that are wrong:

1. Verify before you write. Every technical assertion that enters the skill (endpoint, parameter, command, version, behavior) must trace to something you executed and observed — in this session or an explicitly approved mined one. Can't verify it right now? Either go verify it, or mark it explicitly ("unverified — from memory"). A skill multiplies whatever it contains: verified knowledge compounds, and so do confidently-stated errors. For knowledge skills (content is mostly facts about an external system — API endpoints, parameters, fields, platform behavior), read references/knowledge-skill-grounding.md for the operational version: the authority ladder (observed behavior > machine-readable contract > exercised production code > official docs > memory), evidence-scope annotation, pre-ship doc-example smoke runs, and the audience/Windows portability checklist. A source-grounding audit once found multiple confident contract claims that contradicted evidence already available to the author (methodology Case 9).
2. Treat "impossible / not supported" as a hypothesis, not a conclusion. When a capability seems blocked (an API error wall, a tool that won't connect, a format that won't open), exhaust the observation paths — the UI's own network traffic, an alternative channel, a different documented identifier — before writing "the platform doesn't support this" into a skill. Observed behavior outranks speculative request shapes.
3. Stand on the field's shoulders — retrieve the domain's established best-practices into context BY DEFAULT, before authoring or optimizing a skill's methodology. A skill's methodology is only as good as the knowledge in your context window, not the knowledge latent in your weights: pretraining is lossy, goes stale, and often is not even activated unless the canonical sources are actually pulled in. So the quality ceiling of what you write is your training data + the user's input — unless you deliberately retrieve the subject domain's real prior art. Do it: WebSearch the field's canonical theory / standards / methods, and read any bundled or installed skill in that domain, then fold the load-bearing principles into the skill with attribution. This is a different axis from "Prior Art Research" below — that finds tools/infrastructure to reuse; this grounds the quality of the methodology itself in the discipline's accumulated science. Make it the default action, not something you wait to be asked for: briefly tell the user which field you're pulling from and let them say "skip," but never ship a methodology capped by your memory plus their prompt when 40 years of the field's public work is one search away. Examples: a data-visualization skill must absorb Cleveland & McGill's graphical-perception ranking and Bertin's visual variables (position/length beat color beat text — measured, not aesthetic); a date/time skill must surface the mature libraries and their canonical pitfalls; a persuasion/negotiation skill must retrieve the established frameworks rather than reinvent them from memory. If the canonical knowledge lives only in your weights and never enters context, you are guessing where you could be citing.
4. Preserve before you compress an existing skill. Updating an existing skill is a migration, not a blank-page rewrite. Before the first edit, capture the complete old bundle with the audit tool's snapshot command, or reconstruct it from an explicit Git ref; an arbitrary copy plus a provenance label is not a baseline. Inventory runtime capabilities, trigger contexts, interfaces, references, and eval coverage. Progressive disclosure and concision authorize moving or deduplicating content; they do not authorize silently deleting behavior. After editing, run scripts/audit_skill_regression.py and classify every unmatched old unit. A runtime contract that survives only in evals/, tests, or an unlinked reference is still lost. Do not call the update complete while any candidate is unclassified or any true gap remains unfixed. The same logic governs reversals, not just deletions, and covers any prior commitment — not only the ones carrying a date and a name: overturning a decision already made is a proposal, never a side effect. Say it out loud and get it accepted. A silent rewrite is worse than a silent deletion, because it destroys the artifact and the evidence that could have caught it in one move — and it blinds every downstream reviewer (see #5).