skill-optimizer

Skill Optimizer

Self-evolving skill optimization. Treats SKILL.md as the trainable parameters of a frozen agent. Validation-gated, budget-capped, atomic-versioned.

Based on SkillOpt (arXiv 2605.23904, Microsoft Research, May 2026).

When to invoke this skill

The user wants to:

• Improve an existing skill's execution quality against a benchmark
• Bootstrap a benchmark file for a new skill
• Re-tune a skill after switching target models

Model roles

Three model roles per run, all resolved through Bedrock:

• optimizer — proposes edits from reflection; synthesis tier (Sonnet).
• target — executes the skill on benchmark tasks as an agent would.
• judge — rule judges are deterministic and free; llm judges run on the utility tier (Haiku) by default, synthesis tier when the check needs nuance.

Iron Law

• Validation gating is MANDATORY. Every candidate must clear median-of-3
  • epsilon=0.05 margin against the sel-set before SKILL.md gets rewritten.
• Frontmatter mutation is FORBIDDEN. The optimizer only edits the body. Routing surface (triggers:, brain_first:) stays invariant.
• Bundled skills require explicit opt-in AND an independent held-out set. Skills shipping in the core skillpack cannot be auto-mutated. To rewrite one in place the user passes BOTH --allow-mutate-bundled AND --held-out <path> with at least 5 benchmark-disjoint tasks; without the held-out set the run hard-refuses (exit 2). Drop --allow-mutate-bundled (or pass --no-mutate, the default for the background-cycle phase) to write proposed.md for review instead — no held-out needed for review-only output.
• Bootstrap output requires human review. Both --bootstrap-from-skill and --bootstrap-from-routing write a sentinel; you must review + STRENGTHEN the generated judges, delete the sentinel, and re-run with --bootstrap-reviewed before optimization can use the file.

The pipeline

memex eval skillopt <skill-name> [flags]
  │
  ├── Pre-flight gates
  │     ├── working tree clean (or --force)
  │     ├── benchmark valid + D_sel >= 5 (D17)
  │     ├── cost preflight (D3) — refuses over --max-cost-usd
  │     └── per-skill DB lock (D14)
  │
  ├── Baseline eval on D_sel (sets best_sel_score)
  │
  ├── for epoch in 1..N:
  │     for step in 1..steps_per_epoch:
  │       ├── forward pass: rollouts on D_train batch
  │       ├── backward pass: reflect × 2 (failures + successes per D7)
  │       ├── rank + clip via LR cosine schedule
  │       ├── apply edits (body-only per D5, tagged result per D9)
  │       ├── validation gate: median-of-3 + epsilon=0.05 (D12)
  │       └── if accept: commit via D8 history-intent-first
  │     │
  │     └── slow update (D6) if no improvement this epoch
  │
  └── Final test eval on D_test → run receipt