sbatch-scripts

SLURM sbatch script conventions

When creating or editing sbatch_*.sh under any model's examples/ directory:

Required patterns

1. SCRIPT_DIR resolution — Use scontrol to recover the original script path (SLURM copies scripts to /var/spool/slurmd/):

   if [[ -n "${SLURM_JOB_ID:-}" ]]; then
       SCRIPT_DIR=$(cd "$(dirname "$(scontrol show job "$SLURM_JOB_ID" | grep -oP 'Command=\K\S+')")" && pwd)
   else
       SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
   fi
   

2. Use --rocm for AMD GPU passthrough — Never use --nv (that is for NVIDIA):

   apptainer exec --rocm ...
   

3. Naming convention — Scripts are named sbatch_<task>_amd.sh, not _mi300x.sh or _mi250x.sh. The same rocm7.2.x image covers MI250X (gfx90a), MI300X (gfx942), and MI350X (gfx950).

4. Placeholder partition/account — Use YOUR_PARTITION_HERE and YOUR_ACCOUNT_HERE in #SBATCH directives so users (or agents) know to replace them:

   #SBATCH --partition=YOUR_PARTITION_HERE
   #SBATCH --account=YOUR_ACCOUNT_HERE
   

5. No site-specific paths — including in defaults — Never hardcode user- or cluster-specific paths anywhere in the script, including as default values for env vars. This is the most common mistake: satisfying "paths come from env vars" while still baking a site-specific path into the default.

   Wrong:

   HG_SIF="${HG_SIF:-/shared/username/models/HydraGNN/images/rocm_pytorch.sif}"
   

   Right — use AI4S_SHARED_DIR (set by the user for their site) or leave the default empty and fail with a clear message:

   # Option A: relative to a user-set base dir
   HG_BASE="${AI4S_SHARED_DIR:-}/models/HydraGNN"
   HG_SIF="${HG_SIF:-${HG_BASE}/images/rocm_pytorch.sif}"
   
   # Option B: no default — require the user to set it
   : "${HG_SIF:?Set HG_SIF to your Apptainer SIF path}"
   

   Site-specific values (partition, account, shared dir, scratch paths) belong in the gitignored .cluster-config.yaml / ~/.config/ai4science-studio/cluster.yaml, not in committed scripts. make check enforces this via ai4s_validate.site_leaks.