evaluation

NeMo Evaluator Launcher Assistant

Guide the user through creating NEL YAML configs, running evaluations, and monitoring progress.

Workspace integration

If MODELOPT_WORKSPACE_ROOT is set, use the common skill's workspace-management.md and reuse existing workspaces (this skill is usually the final stage of PTQ → Deploy → Eval; carry any deployment-time patches into deployment.command).

Workflow

- [ ] Step 0: Check workspace (if MODELOPT_WORKSPACE_ROOT set)
- [ ] Step 1: Check `nel` install + existing config; set up `.env` (+ `modelopttools:eval-config` for judge-scored runs)
- [ ] Step 2: Build base config (5-question flow OR shortcut)
- [ ] Step 3: Configure deployment (model path, params, cross-check)
- [ ] Step 4: Fill remaining ??? values
- [ ] Step 5: Confirm tasks (iterative)
- [ ] Step 6: Multi-node (if needed)
- [ ] Step 7: Interceptors (if needed)
- [ ] Step 7.5: Container auth (SLURM private images)
- [ ] Step 8: Dry-run → canary → full run
- [ ] Step 9: Verify completed run

---

nel-next path (Terminal-Bench 2.x, SWE-bench, …) — branch here FIRST

A few agentic AA benchmarks do not run on the currently validated nemo-evaluator-launcher 0.2.6 path (Steps 1–9 don't apply). They run on nel-next (nemo-evaluator[harbor] 0.4.x) — a separate package, CLI (nel eval run), -O overrides, and services/benchmarks/cluster/output schema. If the user asks for one, do not add it to a 0.2.6 evaluation.tasks list — instead:

1. Read references/nel-next.md (shared: venv, schema, AWS creds, architecture, timeout strategy, MLflow, run flow) + the per-benchmark recipe recipes/tasks/aa_next/{terminal_bench_2_1,swebench_verified}.md; start from recipes/examples/example_eval_next.yaml.
2. Isolated nel-next venv: "$SKILL_DIR/scripts/nel-next.sh" --setup-only (keeps 0.2.6 nel untouched).
3. Run modelopttools:eval-config (Step 3b) to write the AWS-sandbox creds + harbor infra rows (${NEL_NEXT_EVAL_IMAGE}, ${HARBOR_*_ECR_REPOSITORY}) into .env; always include the output.export_config.mlflow block.
4. Dry-run → canary → full (nel-next.sh eval run), then push to MLflow — SLURM doesn't auto-export, so run nel-next.sh mlflow-push -r <run_id> -c <cfg> after (config-driven; see references/nel-next.md).