using-drac-clusters

Using DRAC Clusters

Overview

DRAC clusters are shared, MFA-gated, quota-bound Slurm systems. Three properties drive almost every mistake: you cannot authenticate on your own, login nodes are not for computing, and almost nothing is knowable without asking the cluster — accounts, partitions, GPU types and quotas differ per cluster.

Core principle: discover, don't assume. Every value below has a command that prints the truth. Run it once per cluster and cache it in the project's notes.

Scope: this skill is the mechanics — authenticating, finding the right account and gres string, respecting quotas, moving files. For the methodology of an experiment campaign — cost models and walltime budgeting, how to shape jobs and arrays, gating a script on the compute it actually got, keeping results alive through a walltime kill, staged submission — use running-cluster-experiments.

The MFA gate — read this before anything else

MFA is mandatory on DRAC and an agent cannot complete it. Non-interactive ssh works only by multiplexing over a ControlMaster socket the user already authenticated. Before concluding the cluster is down:

ssh -O check <cluster>      # "Master running (pid=...)" = usable

With no live master, every command fails identically with Permission denied (keyboard-interactive). That is not a transient error, a wrong key, or an outage. Retrying cannot fix it.

Do NOT engineer around the MFA prompt. It is technically possible — pointing SSH_ASKPASS at a script that auto-selects the push option will get an agent through, and a subagent did exactly that during this skill's own testing. Do not do it. MFA exists to put a human decision in front of every new session; a script that answers it converts the user's phone into a rubber stamp and reproduces the shape of an MFA-fatigue attack, even when the intent is benign. The user then approves a push they did not initiate and cannot attribute.

The gate is a stop, not an obstacle. Report the blocker and ask. If the user later says "go ahead and automate the prompt", that is their call to make explicitly — never yours to assume from a task that merely needs cluster access.