Gym: Skill for Codex

NeMo Gym Score Triage

Invocation Check

Use this skill when the question is "why is this number different?" rather than "why did this run fail?". Most reported gaps come from comparing unlike things: another code version, another benchmark version, another method, other failure accounting, or ordinary noise. Rule those out before treating the gap as a change in the model.

• If the run crashed, produced no rollouts, or the metrics file is missing, use the nemo-gym-debugging skill first.
• If both runs are comparable and the question becomes which tasks fail and why, hand off to the nemo-gym-blade-analysis skill.
• For the meaning of profiling files and keys, see the nemo-gym-reward-profiling skill.

Collect First

Get these for both sides of the comparison. A reference without a config or a date is a rough target, not a contract.

• Both numbers, and where each came from: document and section, leaderboard page and the date it was read, or a run directory.
• The run directory: rollouts JSONL, *_failures.jsonl, *_materialized_inputs.jsonl, *_aggregate_metrics.json, *_resolved_config.yaml, and the health reports quality_summary.json and rollout_verdicts.jsonl.
• The Gym commit and config:
  • Since #4908, every gym eval run saves its merged config, secrets masked, beside the rollouts as <stem>_resolved_config.yaml. It keeps ${...} interpolations as written, so values read from environment variables (GDPval's JUDGE_MODEL_NAME, for example) are not in it; take those from the launcher's environment or driver.env in resolved-config.yaml. For older runs, rebuild the config with gym env resolve --config <benchmark config> --config <model config> ++key=value, which resolves today's YAML rather than what ran and fails on an unset environment variable without a default (set a dummy value).
  • Plain gym eval run records no commit, so use the launcher's log or git rev-parse HEAD of the checkout that ran it.
  • gym eval submit also writes gym-job.json and the submit config resolved-config.yaml into the run directory. gym_commit in gym-job.json is the Gym that submitted the job (a -dirty suffix means uncommitted changes or untracked files). The job itself runs driver.gym_install.ref from resolved-config.yaml (a branch or tag resolves when each job starts), or the Gym inside the driver container when that is unset. That is the commit that scored the run.
  • Note container or image tags too.
• Model, checkpoint and precision; serving engine and version; reasoning and tool-call parsers; parallelism and GPU type.
• Sampling and limits: temperature, top_p, max output tokens, context length, reasoning effort or mode, chat-template arguments, system prompt.
• Agent harness, judge model, user simulator, tools, timeouts, max steps or turns, and repeats.