evolve-skill

Evolve a skill

Use historical traces to diagnose weaknesses and propose a candidate. Evaluate the candidate by executing the same tasks with both full skill versions and judging the resulting outputs against one unchanged rubric.

1. Diagnose and prepare

Run harvest-sessions, then build-golden <skill> --top 20. Review candidates and label explicit successes positive, failures negative, and ungraded examples skip; run approve-golden <skill>.

Optional dspy-eval <skill> --dataset golden --emit-tasks assesses recorded outputs retrospectively. Its --skill-file changes only the judge rubric. Neither score nor score difference measures how a candidate behaves. Older before/after comparisons using this method are invalid, including those that supplied --skill-file.

Before optimization, reserve self-contained comparison cases outside the training and selection data. Each JSONL row requires:

{"case_id":"decimal-sum","task_input":"Return the sum of 17 and 25 as a JSON object with key total. No tools are needed.","acceptance_criteria":"Return exactly {\"total\":42}, with no other text."}

Replace this illustrative case with tasks relevant to the skill. Include all necessary source/context in the task or use separately reset, isolated fixtures. Historical prompts lacking their repository state are not executable fixtures. Freeze an independent rubric in a text file before seeing either output. Define correctness, applicable procedure, and conciseness on a 0–10 scale.

2. Propose a candidate

python3 distillery/scripts/distiller.py evolve <skill> --optimizer gepa --iterations 5 --save

This step uses paid model calls; follow the user's spending authorization. Keyword fitness is a lexical proxy restricted to positive references or curated expected_output, never failed traces. LLM-judge fitness generates single-turn responses, not tool-using execution. Neither proves agentic improvement. Inspect the diff and size constraints; unchanged output is a null result, not proof of Pareto optimality.