running-cluster-experiments

Running Cluster Experiments

Overview

This is the methodology of experiment campaigns on shared clusters. For the mechanics — MFA, account strings, gres names, quotas, partitions, rsync flags — REQUIRED BACKGROUND: use using-drac-clusters.

Core principle: a scheduler will kill your job at an arbitrary point, and a cluster will hand you less than you asked for. Design for both, or your allocation buys nothing. Every rule below is a consequence.

The expensive failures are never "the code crashed". They are: the job ran perfectly and wrote nothing; the job ran a different experiment than its name says; the allocation was spent before the first useful instruction.

Never compute on a login node — use an interactive allocation

Login nodes are shared by hundreds of people and staff kill offenders. squeue, ls, sacct, diskusage_report, editing, sbatch, file transfer: fine. Anything that loops over data, parses a large file, trains, plots, or runs for more than a few seconds: not fine — including "just this once to check something", and including work you started before realising how big it was.

There is always an alternative, so there is never an excuse. Interactive compute is a normal allocation, not a special favour:

salloc --account=<alloc> --cpus-per-task=4 --mem=16G --time=1:00:00   # shell on a compute node
srun --jobid=<id> --pty bash                                          # shell inside a running job

Short interactive requests usually start in seconds — a measured example: a 1-hour CPU job and a 30-minute GPU job on a busy cluster both began within ~2 minutes, while an 8-hour job on the same cluster waited 13 hours. Asking for less gets you compute faster than not asking at all.

If interactive scheduling is genuinely slow, the answer is sbatch, not the login node.

Build a cost model before you choose a walltime

Measure per-unit cost at the real configuration, on the machine that will run it, then multiply. Not at a smaller model, not at a shorter run, not on a different cluster, not from last quarter's estimate.