overview

Overview

Use the cluster as a shared research instrument. Your job is not just to make code run; it is to request the right resources, avoid repeat work, and leave scarce resources available when you are not using them.

Mental model

• Login nodes are for editing, Git, modules, small installs, and submitting jobs.
• Compute nodes are for Python/R/Stata analysis, simulations, model training, and notebooks.
• Slurm allocates CPUs, memory, GPUs, and time.
• GPFS is shared storage. It likes large files; it dislikes millions of tiny files.
• Modules provide cluster-managed software: Python, R, Git, CUDA, Stata, MATLAB, etc.
• Project space under /gpfs/project/ is for shared research projects.
• Scratch under /gpfs/scratch60/ is temporary working space. Treat it as a staging area, not an archive; remove large intermediates when a project or run is done.

Cluster facts agents should know

Current as of 2026-06-09. Verify live details with sinfo -s (and sinfo -a for restricted partitions) when generating production instructions.

• Scheduler: Slurm. Module system: Lmod (Spack-built; users run module load, never spack).
• Partitions (six; use sinfo -a to see all):
  • default_queue — the Slurm default, but a short, small one: 1h default / 4h max, only 6 nodes (b001–002 + c018–021). Good for quick test jobs; not the place for long or large work.
  • cpunormal — CPU-only production queue, no time limit (c018–021).
  • gpunormal — the general production queue on the GPU-equipped nodes, no time limit, and by far the most capacity (sinfo -s for live node/core counts). Despite the name it is not GPU-only: CPU jobs run here too (request a GPU with --gres=gpu:…, omit it for CPU work). In fact most of the cluster's CPU cores live on these nodes, so large CPU work realistically runs here — but mind your CPU/RAM reservation so you don't strand idle GPUs (see managing jobs). Holds RTX 8000 and A100 nodes, several GPUs per node (sinfo -N -p gpunormal -o "%N %G" for the live map).
  • h100 — 4 H100s per node, two nodes / 8 H100s when healthy. The scarcest GPU resource.
  • build — A40 build nodes (b001–002).
  • mbacourse — access-restricted course partition (you'll see it in sinfo -a but likely cannot submit to it). It shares nodes c001–008 and c018–021 with the queues above.
• cpunormal/gpunormal are the matched "normal" production tier (no time limit); default_queue is just a 4h-capped default on a few nodes. For real, long, or large work prefer cpunormal/gpunormal.
• The partitions overlap on shared hardware, so the cluster can look idle yet your job still pends. The CPU nodes c018–021 are claimed by default_queue, cpunormal, and mbacourse; when a higher-priority partition (e.g. mbacourse) holds them, a default_queue job pends with reason "Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions" even while gpunormal's c001–017 sit mostly idle. Always read the pending reason (squeue --me -o "%T %r") and check idle cores per partition (sinfo -o "%R %C") rather than assuming "the cluster is full."
• The h100 partition is still the scarcest GPU resource. Check live state before targeting it; drained or invalid nodes are unavailable even if they appear in the partition.
• /gpfs/project/ for shared team projects (request via [email protected]). /gpfs/scratch60/$USER/ for scratch (clean it). Each compute node also has local /tmp (~20 GB NVMe), /local (~700 GB NVMe), and /dev/shm (RAM-backed, ~half the node's memory) — all per-node and not auto-cleaned by Slurm. See using the filesystem.
• $HOME is /home/$USER, same as /gpfs/home/$USER.
• KillWait=30: after final SIGTERM, jobs have ~30 seconds before SIGKILL. Too short to rely on for checkpointing.