Gym: Skill for Codex

Add or Update a Harness in NeMo-Gym

The Gym page Add or Update a Harness is the contract and command reference. When it and this skill disagree, the page wins. This skill adds the order of work, the decisions to make before writing code, and the checks a harness must pass before it merges (Step 7). Read the linked sections; do not work from this summary alone.

Words used below follow the page's Terminology: the harness is the agent runtime, the adapter is the Agent Server that wraps it. A pairing is one harness run on one benchmark: the adapter is the agent, the benchmark's Resources Server prepares and verifies the tasks, and a single_agent_turn Environment Server connects the two for each episode.

Step 0: Scope before code

1. Check what already exists. Look for the harness in Built-in agents, Sandboxed harnesses, and under responses_api_agents/, including generic bridges such as harbor_agent that can already launch some harnesses on the legacy /run path. If an adapter exists, the work is an update or a migration, not a new adapter. Read its README and tests first.
2. Pick the path. Use Choose a Path. This skill covers an adapter that runs one agent turn per episode through single_agent_turn. An agent that runs its own multi-turn episode, simulated users, or several verify calls needs its own Environment Server instead.
3. Pick the execution model and read Select an execution mode.
   • Sandboxed harness: the harness runs inside the benchmark's task sandbox, supervised by SandboxSession. Models: pi_agent and codex_agent (CLIs), hermes_agent (a Python harness started through a shim).
   • In-process harness: an SDK or framework that runs inside the Agent Server with its own tools. Model: simple_agent for the seed hook and the session check at the top of responses(). Its close returns no bundle when model-call capture is off, so take the close fallback (a bundle with an observation_capture_failed gap) from hermes_agent. Do not copy langchain_deepagents_agent until #5003 lands: it inherits the hooks, but its responses() does not check the session yet.
   • Host execution is optional. If the adapter offers it, keep it working and test it separately.
4. Pick the capability level. Use Choose the capabilities you need: Evaluation, Observability, Environment tools and task sandboxes, Training. The level decides which later checks apply. Write it down; it goes in the adapter README.
5. Name the first pairing. Choose a benchmark whose Resources Server supports the session contract, and read its README (for example benchmarks/terminal_bench_2_1/README.md) for its run config and the harnesses it lists. Check that the harness's real behavior fits the tasks: a harness that only edits files cannot solve tasks that need commands run. If the benchmark's Resources Server config declares allowed_agents, a new harness is not in it yet; a server that declares none accepts any harness. During development, pass --allow-unsupported-pairing to the command that starts the servers (Step 5). Add the agent to a declared allowed_agents in the same PR only after Step 5 passes, and ask the benchmark's owner to review that line. Do not edit the benchmark's data, prompts, or verifier to fit the harness.
6. Ask the user for what you cannot invent: the pinned harness version and its runtime requirements, a model endpoint and its credentials (kept in env.yaml, never in committed config), a sandbox provider for sandboxed runs, and approval for downloads such as task images or benchmark data.