skill-tuning-loop

Skill Tuning Loop

A skill earning its place in the lacquer (per skill-authoring-standard) is a judgment call today. This loop adds a second, empirical bar: does a proposed edit measurably reduce the friction real sessions hit, without regressing what already works? Run it via the skill-tuning-loop Workflow — never edit a skill's behavior-affecting content based on mined evidence without going through the validation gate below.

When to run this

• A skill keeps needing the same correction across unrelated sessions — that's a signal worth mining, not a one-off to shrug off.
• Before accepting an automation-proposed edit (see the continuous-tuning design this skill's workflow implements) — the gate is what makes an unattended proposal safe to merge.
• Never on every session, and never on a single session's evidence — one grumpy correction is noise, not a pattern. Require independent evidence from multiple sessions before proposing a change.

The loop (implemented in the skill-tuning-loop Workflow)

1. Mine. Search recorded sessions (via the agentsview MCP tools) for invocations of the target skill plus friction phrases ("no that's wrong", "actually", "don't do that"). Pull the surrounding messages for anything that looks skill-relevant.
2. Reflect. Have an agent read the current SKILL.md against that evidence and name concrete, recurring patterns — what the skill says (or omits) and what went wrong because of it. Require ≥2-3 independent sessions per pattern; thin evidence gets flagged, not acted on.
3. Propose. Generate a bounded edit — add/replace the few lines implicated by the evidence, not a rewrite — that still satisfies skill-authoring-standard (trigger-oriented description, instruction over exposition, no padding).
4. Validate. Run the same small held-out task set (references/eval-cases.md if the skill has one, else 3 representative tasks derived from its description) against the old and new skill text in parallel, judged by an independent agent running on a stronger model than the one that drafted the proposal — the same posture as advisor-checkpoint, spending the expensive model at the one load-bearing moment (the ACCEPT/REJECT verdict) rather than throughout. Accept only if the new version doesn't regress any case and strictly improves at least one. A good eval case is a task plus an explicit, checkable success criterion — not "handles it well," but the specific behavior that counts as a pass — and the set should include at least one edge or ambiguous case, not just the obvious path. If you're authoring references/eval-cases.md by hand for a skill, hold it to that same bar.
5. Report, don't merge. A REJECT verdict ends the loop — log the rejected proposal and reasoning so a future pass doesn't re-propose it without new evidence. An ACCEPT verdict produces a diff and its evidence (session excerpts, before/after eval scores) for a PR — a human merges it. This loop never commits, pushes, or merges on its own.