Gym: Skill for Claude Code

Add Benchmark to NeMo-Gym

Determine Integration Type

Before starting, determine which type of benchmark you're adding:

Native benchmark — verification logic implemented directly in a Gym resources server:

• Resources server implements verify() with reward logic
• Agent server orchestrates model calls (use simple_agent for single-turn, or custom agent for multi-turn)
• Example: code_gen, instruction_following, math_with_judge

External benchmark — wrapping a 3rd-party library that has its own orchestration:

• Integrate at the agent server level (not resources server)
• Agent's /run endpoint wraps the external library
• Pre-process from Gym schema to library input, post-process back to BaseVerifyResponse
• Reproduce publicly reported numbers with the original repo first, then reproduce again after Gym integration
• Add the dependency in requirements.txt

Workflow

Step 1: Scaffold the server

Run gym env init to generate the directory structure:

gym env init --resources-server my_benchmark

This creates:

resources_servers/my_benchmark/
├── app.py              # Server template
├── configs/my_benchmark.yaml
├── data/.gitignore
├── tests/test_app.py
├── requirements.txt
└── README.md

For external benchmarks, create the agent server manually under responses_api_agents/my_agent/ with the same structure.

Step 2: Prepare data

Convert your source dataset to Gym JSONL format. Each line must have responses_create_params.input (OpenAI message format). Task-specific verification data goes in verifier_metadata.

{
  "responses_create_params": {
    "input": [
      {"role": "system", "content": "System prompt"},
      {"role": "user", "content": "Problem statement"}
    ]
  },
  "verifier_metadata": {
    "test_cases": [{"input": "...", "expected_output": "..."}],
    "task_id": "unique_id"
  }
}

Data conversion: Write conversion scripts in the source repo (e.g. your dataset repository), not in NeMo-Gym. Prompt files also belong in the source repo. Exception: when there is no external source repo. See references/patterns.md § "Data Conversion Script Pattern".