opik-evaluate

LLM Evaluation

Help users build, audit, and improve evaluation systems for LLM pipelines.

Where to Start

Have an existing eval pipeline? Start with an eval audit to surface problems: missing error analysis, unvalidated judges, vanity metrics. See the eval-audit reference.

Starting from scratch? Begin with error analysis on real traces. If no production data exists, generate synthetic data first. See the error-analysis and generate-synthetic-data references.

Test Suites

Test suites are the primary way to test agents in Opik. They combine test items with string assertions checked by an LLM judge, plus execution policies for multi-run reliability testing. Available in both Python and TypeScript SDKs.

Python:

import opik

client = opik.Opik()
suite = client.get_or_create_test_suite(
    name="my-agent-suite",
    global_assertions=["Response is factually accurate", "Response is professional"],
    global_execution_policy={"runs_per_item": 3, "pass_threshold": 2},
    project_name="my-project",
)

suite.insert([
    {"data": {"input": "What is the capital of France?"}, "assertions": ["Mentions Paris"]},
])

results = opik.run_tests(
    test_suite=suite,
    task=lambda item: {"output": my_agent(item["input"])},
    model="gpt-4o",
)
assert results.all_items_passed

TypeScript:

import { Opik, runTests } from "opik";

const client = new Opik();
const suite = await client.getOrCreateTestSuite({
  name: "my-agent-suite",
  globalAssertions: ["Response is factually accurate", "Response is professional"],
  globalExecutionPolicy: { runsPerItem: 3, passThreshold: 2 },
  projectName: "my-project",
});

await suite.insert([
  { data: { input: "What is the capital of France?" }, assertions: ["Mentions Paris"] },
]);

const results = await runTests({
  testSuite: suite,
  task: async (item) => ({ input: item.input, output: await myAgent(item.input as string) }),
  model: "gpt-4o",
});
if (!results.allItemsPassed) process.exit(1);