managing-jobs

Managing Jobs

Rule: test small, request explicitly, monitor results, then scale.

Before you submit a job — or recommend one to a human — ask "am I being a good citizen?" and answer it: right partition for the work, requested only what it uses (right-size --mem/--cpus/--time), not stranding idle GPUs with a CPU/RAM reservation, not storming GPFS with tiny files, and cleaning up after. The concrete how-to-answer for each is the citizenship self-check in overview. When you advise rather than submit, tell the human which checks you ran.

Basic commands

sbatch job.sh                    # submit batch job
squeue -u $USER                  # current jobs
sacct -j 12345                   # completed job accounting
scancel 12345                    # cancel your job
scontrol show job 12345          # detailed job state
sinfo -s                         # partition summary

Common job states: R is running, PD is pending, and CG is completing. If a job stays pending unusually long, inspect the reason instead of just waiting; the request may not fit available nodes, memory, GPUs, partition limits, or time limits.

squeue --me --start
scontrol show job JOBID

Choosing a partition

default_queue is the Slurm default, but it is a short, small queue (1h default / 4h max, only 6 nodes) and those nodes are shared with — and outranked by — the restricted mbacourse partition. So a default_queue job can pend with "Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions" even when the cluster is mostly idle, because the idle cores are on partitions it doesn't target.

Before assuming "the cluster is full," look:

squeue --me -o "%.10i %.9P %.2t %.10M %r"   # %r = pending reason — read it
sinfo -o "%R %C"                            # idle cores per partition (A/I/O/T)

Pick the partition that fits the work: