pi-skillopt

PI SkillOpt Operator

Use this skill when the user wants the agent to operate the SkillOpt repository: run experiments, evaluate skills, configure local models, inspect outputs, or extend the repo with new benchmarks/backends.

Core repo facts

• SkillOpt optimizes a skill document / system prompt, not model weights.
• The repo supports a generic local OpenAI-compatible backend named openai_compat.
• The local example model config is configs/dotnetdebug/local_mitko.yaml.
• That config uses model mitko at http://localhost:8000/v1 for both optimizer and target.
• The runnable sample benchmark is dotnetdebug.
• The sample dataset is data/dotnetdebug/tasks.json.
• The seed skill is skillopt/envs/dotnetdebug/skills/initial.md.
• Training entry point: scripts/train.py.
• Eval-only entry point: scripts/eval_only.py.

Default workflow

1. Identify the user goal:
   • run a SkillOpt training job
   • evaluate an existing skill
   • inspect outputs from a previous run
   • add or modify a benchmark
   • add or modify a backend
2. Confirm the benchmark, backend, target model, and desired output directory.
3. Prefer a small smoke test first before a larger run.
4. Use existing configs when possible instead of inventing new ones.
5. After a run, report:
   • exact command used
   • output directory
   • best skill path
   • headline metrics
   • next recommended action

Local model workflow

When the user asks to use a local model or mentions mitko, localhost:8000, or an OpenAI-compatible endpoint:

• Prefer model.backend: openai_compat.
• Set both optimizer_backend and target_backend to openai_compat unless the user explicitly wants a mixed setup.
• Use configs/dotnetdebug/local_mitko.yaml when the task is the built-in dotnet debugging example.
• Keep smoke tests small, for example:
  • train.num_epochs=1
  • train.batch_size=2
  • gradient.analyst_workers=1
  • gradient.minibatch_size=2
  • env.workers=1
  • env.limit=2