deploy-spur

Deploy Spur Cluster

Spur is an AI-native job scheduler with these daemons:

• spurctld — controller / scheduler / Raft consensus (1 instance, or ≥ 3 for HA). Also serves accounting (sacct/fairshare, backed by PostgreSQL) in-process on its own gRPC port whenever [accounting].database_url is set — there is no separate accounting daemon. Only Postgres itself is a distinct service, on ACCT_HOST (default: first controller; may be a dedicated node).
• spurd — node agent, runs on every compute host

Pre-merge Spur builds additionally shipped a standalone spurdbd accounting daemon; upstream folded it into spurctld. If you're upgrading a cluster that still has spurdbd.service active, see Step 5b: migrating off a standalone spurdbd below — don't skip it, or you'll end up with two accounting paths fighting over the same Postgres.

This skill stands a cluster up with only SSH + bash on the targets. It is the standalone equivalent of ansible/ and reaches the same end state: daemons run as systemd services (survive reboot), Slurm-compatible CLI names are symlinked, and accounting is optional (default on). One flow covers all four topologies — only the host list, Raft topology, and which hosts run an agent differ. A rolling-upgrade flow (Step 12) upgrades an already-running cluster one host at a time instead of bouncing everything at once.

Defaults (override only if the user asks):

Var	Default
SPUR_HOME	/root/spur
SPUR_INSTALL_DIR	/root/.local/bin
SPUR_VERSION	latest (passed to install.sh; or nightly / vX.Y.Z)
SPUR_BINARY_SRC	(empty) — local dir with pre-built spur/spurctld/spurd and optionally spur_mpi_pmix.so; used when set, else install.sh
SPUR_MPI_PLUGIN_DIR	/usr/lib/spur — where spur_mpi_pmix.so is installed on agents (matches spurd default)
SPUR_CONTROLLER_PORT	6817
SPUR_AGENT_PORT	6818
SPUR_RAFT_PORT	6821 (hardcoded inside spurctld; cannot be changed via CLI)
ACCT_DB_PORT	5432 (PostgreSQL)
SPUR_CLUSTER_NAME	spur-cluster
SPUR_LOG_LEVEL	info
SPUR_WIPE_STATE	false (preserve Raft state so re-runs/upgrades are non-destructive; set true for a fresh install or intentional Raft reinit)
ACCOUNTING	true (deploy PostgreSQL; accounting is served by spurctld itself; set false to skip)
ACCT_DB_NAME / ACCT_DB_USER / ACCT_DB_PASSWORD	spur / spur / spur
TRANSPORT	direct (LAN) — or wireguard for an encrypted mesh
ROLLING_BATCH_SIZE	1 (agents upgraded per batch in Step 12; controllers are always one at a time)
SSH user	root (unless the user specifies otherwise)

Every command below is written to work whether the SSH user is root or a non-root user with (passwordless) sudo — confirm sudo -n true succeeds during Step 1's preflight. SPUR_HOME/SPUR_INSTALL_DIR default under /root, which is mode 0700: a non-root user cannot even cd/execute/test -x into it, let alone write there, and no chown of a subdirectory fixes this (the block is on traversing /root itself). So:

• Every remote command that reads/writes under /root, /etc/systemd/system, or runs spur/sbatch/sacct/etc. must be sudo-prefixed — not just the file-writing steps. This applies uniformly (as root, sudo is a harmless no-op).
• scp and heredoc redirects (cat > /path <<EOF) run as the plain SSH user and cannot land a file directly under /root even with the SSH user later sudo-reading it. Copy to /tmp first, then sudo install/sudo mv it into place.
• Alternatively, set SPUR_INSTALL_DIR/SPUR_HOME to a world-traversable path (e.g. /opt/spur) up front to sidestep all of this — but then every install/sudo note below is still harmless, just unnecessary.

Password-based SSH works too — every ssh/scp command in this skill is a plain invocation with no auth-method assumptions baked in, so if key-based auth isn't set up, prefix each one with sshpass -p "$SSH_PASSWORD" ssh -o StrictHostKeyChecking=no ... (and the scp equivalent). Do not add -o BatchMode=yes anywhere — it disables SSH's password prompt outright and silently breaks password auth even with sshpass supplying the answer.