opik-optimize

Optimize — Improve a Prompt Against Data

Definition of done: an optimized prompt whose score on the metric beats the baseline on data it was not tuned on, the optimization run link in Opik, the cost of getting there, and the winner saved as a new prompt version (when the prompt lives in the library) — with the swap into code left as the next step. If the optimization can't run within a stated budget, stop at the first genuine blocker and return exactly one next step. A prompt that scores higher only on its own training items is not an improvement.

Operate: measure the baseline first, state the budget before spending it, hold data out, pick the algorithm for the failure you see, save the winner where it can be versioned — and change no application code. The only file this skill writes is a runner outside the repo; the prompt is saved to Opik, not into the codebase.

Inputs

The entry point is /opik-optimize <prompt-name> (a prompt-library prompt), /opik-optimize <path or function> (a prompt in code), or /opik-optimize (find the system prompt in this repo). Infer the rest; treat these as optional overrides:

• dataset (default: existing dataset for the project → export the regression suite → build from traces) · metric (default: heuristic on expected_output if present, else one binary judge) · algorithm (default: MetaPromptOptimizer) · budget (default: n_samples=50, max_trials=10) · model · validation split (default: hold out 20%).

Ask only at a genuine, non-inferable blocker (see Blockers).

Activation — the only in-scope work

1. Resolve the prompt

• Library: client.get_chat_prompt(name) (or get_prompt for a text prompt). Note the current version — that is the baseline.
• Code: grep for the system prompt / messages=[...]; read it verbatim. Note where it lives; you will not edit it.
• Trace: the llm span's input.messages on a representative trace.

The optimizer's opik_optimizer.ChatPrompt is a different class from the library's opik.ChatPrompt — build it from the raw messages yourself:

from opik_optimizer import ChatPrompt
prompt = ChatPrompt(name="<name>", system="<system text>", user="{question}")   # or messages=[...]; {var} names must match dataset keys