evaluate-environments

Evaluate Tasksets

Goal

Set up an evaluation for a taskset in the correct way to reproduce results from others or evaluate a model and harness combination on a given taskset.

Canonical path

Use the eval entrypoint

uv run eval <MY_ENV>

Core workflow

1. Resolve and validate config without model calls:

uv run eval <MY_ENV> --dry-run

1. Run model-free gold validation when the taskset implements validate:

uv run validate <MY_ENV> --runtime.type subprocess

1. Do a small run to see whether it works correctly:

uv run eval <MY_ENV> -m deepseek/deepseek-v4-flash -n 3 -r 1

1. Inspect successful, zero-reward, and errored traces.
2. Scale only after task loading, harness capability, runtime lifecycle, and scoring are correct.

When the user requests a full run, do not restrict the number of tasks. Ask for the appropriate harness to use (if not specified)

IDs and plugin resolution

A plugin id names an installed package (e.g. my-taskset); verifiers imports it and never installs anything itself.

The leading ID is shorthand for --env.taskset.id. A harness belongs to an agent — --env.agent.harness.* on the single-agent env, --env.<agent>.harness.* on a multi-agent one (there is no run-level --harness.*):

uv run eval my-task-v1 --env.agent.harness.id codex --env.agent.runtime.type prime

The env — the control flow between agents — owns the whole [env] block. Empty --env.id keeps the taskset's own story (its exported Env subclass, else the single-agent env); --env.id pairs a reusable env with any taskset, its knobs typed under --env.*:

uv run eval my-task-v1 --env.id best-of-n --env.n 8      # pass@k / rejection sampling
uv run eval my-task-v1 --env.id agentic-judge \
  --env.judge.runtime.type docker                           # a judge agent verifies each attempt in a sandbox

Disabling tools

Almost every harness comes with a disabled_tools list, which can be used to disable one or multiple tools: