skillopt-setup

SkillOpt Setup

Goal

Set up and operate a local SkillOpt workflow that improves one Agent Skill at a time while preserving frontmatter, validation, and public-proof boundaries.

When to use

• Installing or checking Microsoft SkillOpt for this repo.
• Preparing SkillOpt train/val/test data from skill-evals/.
• Configuring provider-backed, hybrid Codex CLI, or exploratory all-Codex CLI execution.
• Running SkillOpt for a target SKILL.md.
• Reviewing, summarizing, or importing best_skill.md.
• Creating public run evidence without committing raw transcripts.

When not to use

• Ordinary skill authoring or manual SKILL.md review.
• Trigger-description tuning only.
• Repo validation without SkillOpt.
• Directly implementing an optimized skill before review.

Inputs to inspect

• Target skill path under skills/ or incubator/skills/.
• skill-evals/<skill>/README.md, cases/, fixtures/, expected/, rubric.md, and runs/.
• Current SKILL.md frontmatter and body.
• .gitignore and AGENTS.md safety rules.
• Local .agents/tools/SkillOpt state, if present.
• Local .agents/ artifact audit when mining prior setup work or troubleshooting stale workspaces.
• Local uv availability and local Python 3.10+ compatibility.
• Local Codex CLI availability and login state if Codex mode is requested.
• Visual assertion cases, render-capable requirements, and draw.io Desktop CLI availability when target evals need PNG/SVG artifacts.

Workflow

Guide the user through setup as a short wizard. Ask one decision at a time unless the user already supplied the answer; do not dump every command or option before the relevant step.

1. Wizard step: target. Identify exactly one target skill and whether it is incubator or promoted. If missing or ambiguous, ask which skill should be optimized.
2. Wizard step: existing setup. Immediately inspect .agents/tools/SkillOpt, .agents/tools/SkillOpt.commit, and .agents/skillopt-work. If any exist, ask whether to remove the current local setup or reuse/update it before dry-run or production setup. Cleanup is global to the local SkillOpt setup and must not remove .agents/skills/.
3. If the user chooses cleanup, run setup-skillopt-local.mjs --cleanup-only --approved yourself before setup. Do not present cleanup as a copy-paste command.
4. When the user asks to move prior .agents/ learnings into the skill, run audit-skillopt-local-artifacts.mjs first. Promote only sanitized scripts, references, templates, eval cases, or curated summaries; never promote raw clones, installed skill copies, data splits, run outputs, transcripts, or readiness diagnostics.
5. Wizard step: setup goal. Explain the recommended branches:

• easiest no-provider path: codex-cli-all, exploratory, uses Codex CLI login for rollouts, semantic judging, and adapter-managed reflection; keep slow update and meta skill disabled in this mode because those upstream epoch-boundary mechanisms call the provider-backed optimizer path,
• best official-parity path: hybrid-codex-target or native-provider, provider-backed optimizer/reflection, requires credentials and model pins.

1. Wizard step: Python. Prefer uv. If uv is missing, ask whether to install uv or explicitly use compatible local Python 3.10+.
2. Wizard step: data quality. Run readiness or split preparation early enough to report positive, validation, and test counts. Official-parity proof needs at least 20 positive cases, 5 validation cases, and 5 test cases; otherwise classify the run as exploratory or blocked for proof.
3. Wizard step: execution and visual readiness. Require the bounded strict-config capability probe for every active Codex target, judge, or reflection role, including text-only cases. If any cases declare visual_assertions, also check visualArtifactReadiness, generated tool_rollout_for_visual_assertions, and visual_eval_policy. Visual Codex cases may use bounded file edits and shell commands only under the enforced read-isolated rollout permission profile, for copied helper scripts, draw.io XML, validation, and requested PNG/SVG exports; non-visual Codex rollouts remain text-only with no workspace read or write grant. If draw.io CLI is missing, use the generated data-text-only split and report that full visual proof still requires the renderer. Native-provider auto mode must also select data-text-only because provider chat targets cannot create local artifacts.
4. Wizard step: best-practice configuration. For official-parity, require provider credential presence plus SKILLOPT_OPTIMIZER_MODEL, SKILLOPT_TARGET_MODEL, and judge model pins. For codex-cli-all, preserve exploratory defaults, require slow update/meta skill to stay disabled, and report which upstream provider-backed behavior is bypassed.
5. Wizard step: dry-run. Ask whether the user wants a dry-run first unless already answered. If yes, run setup without --approved, report the dry-run result only, and ask whether to continue. Do not show production setup commands or SkillOpt training commands after dry-run.
6. Wizard step: production setup. If the user skips dry-run or approves continuation, run production-grade setup with --approved, using .agents/ as the persistent workspace and passing --existing-setup-choice reuse when reuse was chosen.
7. Prepare or update the ignored SkillOpt workspace, split JSON, local adapter, target manifest, and mode/profile config.
8. When the user wants a guaranteed training-ready setup, use --strict-training-ready; block rather than hand off a training command if credentials, model pins, Codex probe, visual artifact readiness, adapter patches, or refreshed target manifest checks are missing.
9. After successful production setup, recommend the paste-ready new-terminal SkillOpt command. It must stream logs, print explicit success/failure, run artifact verification, show a compact summary, and run best_skill.md dry-run adoption preview.
10. Include manual rerun commands with short descriptions for artifact verification, run summary, dry-run adoption preview, eval-only evaluation, and optional WebUI.
11. Offer current-session execution only as an explicit option: Should I run SkillOpt training for <target-skill> in this agent session anyway?
12. Inspect best_skill.md, diff it against the original skill body, validate adoption gates, ask before tracked writes, and save only curated public evidence under skill-evals/<target>/runs/.