opik-compare

Compare — Candidate vs Baseline Over a Suite

Definition of done: two experiments on the same suite — a baseline and a candidate — read back item by item, with the per-metric deltas, the cases that went pass → fail (regressions) and fail → pass (fixes), the worst rows, a note on whether the runs are comparable, and the Opik compare-view link. If only one run exists, the done state is "baseline created — rerun after the change". If the suite can't be run or read, stop at the first genuine blocker and return exactly one next step. Aggregate scores alone are not a comparison; a verdict is not this skill's job.

Operate: run the candidate the same way the baseline was run, read both back from Opik rather than from the run's console output, name the specific cases that changed — and change no application code. The only file this skill writes is a throwaway runner outside the repo.

Inputs

The entry point is /opik-compare <suite> (run the suite now as the candidate, compare against the latest prior run), /opik-compare <experiment-A> <experiment-B> (read two existing runs, no new run), or /opik-compare right after /opik-test (that suite). Infer the rest; treat these as optional overrides:

• suite (default: <project>-regressions, or the one /opik-test just wrote to) · baseline (default: the most recent earlier experiment on the suite) · candidate name (default: candidate-<short git sha>) · judge model for assertions (default: the suite's / the baseline's) · runs per item (default: the suite's policy).

Ask only at a genuine, non-inferable blocker (see Blockers).

Activation — the only in-scope work

1. Resolve the suite and the baseline

import opik
client = opik.Opik()

suite = client.get_test_suite(name="<suite>", project_name="<project>")
prior = client.get_test_suite_experiments(name="<suite>", project_name="<project>")  # newest first is not guaranteed — sort by created_at yourself

Baseline = the most recent prior experiment on this suite, unless the user names one. No prior experiment → this run is the baseline (step 3 still runs; status baseline_created). Two explicit experiments → skip step 3, go to step 4.