deploy-slurm-cluster

Deploy a Slurm GPU cluster

Preconditions

• Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver installs may reboot them; no active users or workloads).
• SSH access from the provisioning machine to every host as a sudo-capable user.
• Run everything from the repository root.

Procedure

1. Prepare the environment and verify it:

   git submodule update --init --recursive
   ./scripts/setup.sh
   cp -r config.example config
   

2. Edit config/inventory: put the controller under [slurm-master] and compute nodes under [slurm-node] (a single machine can be both). Set the connection user in [all:vars] if not root.

3. Preflight — must pass before deploying:

   python3 scripts/validation/deepops_doctor.py --remote --json
   

   Fix anything in failures (each check's detail says how) and rerun.

4. Deploy:

   ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml
   

   This installs NVIDIA drivers, builds and configures Slurm, and sets up munge, NFS, and node health checks. Expect roughly 30–60 minutes on a first run.

5. Validate on a cluster node — the success signal is this, not the play recap:

   python3 scripts/validation/validate_slurm.py --json
   

   Require "ok": true with gpu_job_ok: true and nodes_unavailable: 0.

Failure branches

• Playbook fails on a transient error (mirror timeout, apt lock, network blip): rerun the same playbook; it is idempotent. A converged rerun ends with changed=0.
• nvidia-smi works in the validator's srun job but "fails" over SSH: that is the login GPU-hide behavior, not an error (see AGENTS.md gotchas).
• gpu_job_ok: false with driver errors: follow skills/diagnose-driver-install/.
• Node shows down or drained in node_states: check scontrol show node <name> for the reason; after fixing, resume with scontrol update nodename=<name> state=resume.
• Wrong or stale host facts after reprovisioning a node: rerun with --flush-cache.