agent-optimization

Agent Optimization

Improve one Test Agent through an evidence → hypothesis → Candidate → evaluation → accept or rollback loop. Use public Statements, scores, and Test Traces as black-box feedback. Delegate every evaluation to an agent-evaluation subagent; never run or score the Test Agent directly.

Before you start

If the request does not identify the Test Agent, frozen Benchmark, desired target score, positive Run count, and round limit, ask for the missing inputs. When they are already supplied, proceed without asking the user to restate them.

Goal and contract

Require an explicit Test Agent, a frozen Benchmark with a complete valid Formal Baseline, a desired target score, a positive runs value, and a positive round limit. The Benchmark's benchmark_config.toml must say status = "published"; a draft Benchmark is still being built and is not frozen, and a failed one never finished calibrating, so stop and explain in either case. runs is the number of Runs per Case for every Candidate in this optimization Session. Freeze it for the Session; do not infer it from benchmark_config.toml or the Formal Baseline. Read the evaluation (provider, model_id, thinking_level) from the complete Evaluation that matches the current Agent State; do not require the user to repeat it. An Evaluation without any part of this runtime is incomplete and cannot be used as a Reference. The top-level Session must provide run_subagent, and the current Agent must have the agent-evaluation Skill. If a prerequisite is missing, stop and explain what is needed. Do not create the missing Agent, Benchmark, or Baseline, and do not evaluate the Test Agent directly.

A Reference is the Agent State currently kept as best, together with its complete Evaluation on the frozen Benchmark.

Each round starts from the Reference and tests a bounded, general Candidate. Evaluate every Candidate on the frozen Case set with the requested runs count and the Reference evaluation runtime. The initial Formal Baseline has one Run per Case; do not rerun or backfill it to the requested count. Compare each Candidate's stored top-level average directly with the current Reference score even when their Run counts differ. Accept the Candidate only when the change is admissible, its Evaluation is complete and valid, and its top-level score is strictly higher than the Reference Evaluation's score. An accepted Candidate and its Evaluation become the next Reference; otherwise restore the previous Reference. Stop early when the Reference reaches the desired target; otherwise run no more than the requested number of complete valid Candidate rounds.