opik-evaluate

Evaluate — Build an Evaluation and Run It

Definition of done: an experiment in Opik with scores, its link, and a score summary the user can act on — produced by running the user's app (or prompt) over a set of cases with scoring that targets real failure modes. If the evaluation can't be built or run, stop at the first genuine blocker and return exactly one next step. A dataset with no run, a judge prompt with no scores, or a plan in prose is not success.

Operate: ground the cases in what actually goes wrong, score with code before judges, judge one failure mode at a time, run once end to end, read the scores back from Opik — and change no application code. The only file this skill writes is a runner outside the repo.

Inputs

The entry point is /opik-evaluate (evaluate the app in this repo), /opik-evaluate <what> ("the RAG answers", "the refund flow"), or /opik-evaluate <dataset-or-suite> (run against existing cases). Infer the rest; treat these as optional overrides:

• cases source (default: existing suite/dataset → recent traces → synthetic) · scoring (default: heuristics where an expected output exists, else one judge per failure mode) · approach (default: test suite for agents; dataset + evaluate() when you need built-in metrics or exact matching) · judge model · sample size (default: 20–50 items).

Ask only at a genuine, non-inferable blocker (see Blockers).

Activation — the only in-scope work

1. Resolve the target

Confirm Opik is reachable (~/.opik.config or OPIK_API_KEY; otherwise → Blocker). Find the entrypoint to evaluate (the function a trace's root span names, or the one the user points at). Check what already exists — client.get_test_suites(project_name=…), client.get_datasets() — and reuse before creating.

Have an eval already? Audit it first (references/eval-audit.md): unvalidated judges, no error analysis, vanity metrics. Fix the worst gap, then run.

2. Ground it in failures (error analysis)

Read real traces before writing any scorer — client.search_traces(project_name=…, max_results=100) (errors, low scores, long durations first). Categorize what goes wrong and how often (references/error-analysis.md). No traces yet → generate cases (references/generate-synthetic-data.md) and say so in the report.