slurm-script

Slurm Scripts

This is about what goes in the request and how to read what comes back. For running a Spack package through JARVIS instead of a hand-written script, see cluster-run.

Size the request against a real machine

Look before guessing. clio-node-hardware:get_cpu_info gives core counts, get_memory_info gives RAM, get_gpu_info gives GPUs. Then clio-slurm:slurm_cluster gives the partitions and their limits.

Hardware tools describe the MCP host, not necessarily the allocated compute nodes. Use the target partition/node records as the scheduling authority. Invalid requests may be rejected at submission or remain pending; inspect the actual scheduler reason instead of assuming one outcome.

The shape of a batch script

#!/bin/bash
#SBATCH --job-name=run1
#SBATCH --partition=<from slurm_cluster>
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=1
#SBATCH --mem=32G
#SBATCH --time=02:00:00
#SBATCH --output=run1-%j.out
#SBATCH --error=run1-%j.err

srun ./my_binary

Points that matter more than they look:

• --time is a hard kill, not an estimate. The job is terminated at the limit, mid-write if necessary. But a shorter limit schedules sooner, because it fits in backfill gaps. Ask for what the run needs plus margin, not the partition maximum.
• --ntasks vs --cpus-per-task. Tasks are MPI ranks; cpus-per-task is threads within a rank. An MPI+OpenMP code that sets both wrong runs at a fraction of its speed while looking healthy.
• --mem is per node; --mem-per-cpu is per allocated CPU. Setting both is an error. Getting --mem wrong is the most common cause of a job killed partway through with no obvious message.
• %j in output paths expands to the job ID. For arrays use %A_%a (array job ID and task index), for example run1-%A_%a.out.
• srun inside the script, not a bare call. It inherits the allocation; running the binary directly gets you one rank on one node regardless of what was requested.