skill-creator

Skill Creator

A skill for creating new skills and iteratively improving them.

Overview

At a high level, the process of creating a skill goes like this:

• Decide what you want the skill to do and roughly how it should do it
• Write a draft of the skill
• Create a few test prompts and run claude-with-access-to-the-skill on them
• Help the user evaluate the results both qualitatively and quantitatively
• While the runs happen in the background, draft quantitative evals
• Use the eval-viewer/generate_review.py script to show results
• Rewrite the skill based on feedback from evaluation
• Repeat until satisfied
• Expand the test set and try again at larger scale

Your job when using this skill is to figure out where the user is in this process and help them progress through these stages.

How to Use This Skill

When the User is Starting Out

If they say "I want to make a skill for X":

• Help narrow down what they mean
• Write a draft
• Write test cases
• Figure out how they want to evaluate
• Run all the prompts
• Iterate based on feedback

When the User Already Has a Draft

Go straight to the eval/iterate part of the loop.

Being Flexible

If the user says "I don't need to run a bunch of evaluations, just vibe with me", that's fine too. Adjust your approach.

After the Skill is Done

Optionally run the skill description improver to optimize triggering accuracy.

---

Communicating with the User

The skill creator is used by people with varying familiarity with technical jargon.

Pay attention to context cues to understand how to phrase your communication:

• "evaluation" and "benchmark" are generally OK
• For "JSON" and "assertion", wait for clear signals the user knows what those are before using them without explanation
• It's OK to briefly explain terms if you're unsure

---

Creating a Skill

Step 1: Capture Intent

Start by understanding the user's intent. If the conversation already contains a workflow to capture (e.g., "turn this into a skill"), extract answers from the history first.