Gym: Skill for Codex

Nemo Gym Debugging

Invocation Check

Use this skill when something failed or looks suspicious in a Nemo Gym run. If the task is adding a new env, use the add-benchmark skill; if it is changing profiling behavior, use the nemo-gym-reward-profiling skill.

Debug by classification, not by guessing. The first goal is to decide whether the issue is:

• infra: Slurm, Ray, container, filesystem, network, ports
• model serving: vLLM startup/readiness/throughput
• config: wrong config bundle, missing agent, wrong extra args
• data/schema: JSONL fields do not match verifier/resource server expectations
• verifier/runtime: resource server exception or malformed verify response
• cache/resume: stale materialized inputs or partial rollout output
• throughput/resources: concurrency too high, judge bottleneck, tool/sandbox latency

Debug Order

1. Check Slurm/Ray job state and logs.
2. Check vLLM readiness and /models availability.
3. Check Gym server readiness: all expected servers started.
4. Check tool routing if the env uses tools; check sandbox readiness only if a sandbox is configured.
5. Check materialized inputs and source data timestamps.
6. Check rollout output and profiling/metrics output counts.
7. Inspect the first real verifier exception, not shutdown noise.
8. Compare failing row schema against the resource server request model.

High-Value Suspects

• If data changed and resume_from_cache was enabled, stale materialized inputs are a first-class suspect.
• If rollout output has a few rows and profiling is empty, inspect verifier errors and partial-output cache.
• If all servers are ready but verifier returns 422/500, inspect request body schema before debugging infra.
• If tool envs hang or partially work, check tool ownership/loading before changing model settings; check sandbox readiness only when a sandbox is actually part of the env.
• If tool-call rows fail before generation with vLLM grammar/schema errors, read references/vllm-tool-call-schema-checks.md and run a static tool-schema check before changing Gym wrappers.
• If logs only show nested "inner server" 500s without the real provider/verifier body, first enable existing request-boundary visibility with ++global_aiohttp_client_request_debug=True. Read references/request-boundary-visibility.md before changing code.