init-cluster

Initialize local cluster configuration

Detect the current cluster environment and create a .cluster-config.yaml file with site-specific settings. This file is gitignored and never committed.

Step 0 — Check if config already exists

Check both locations in order:

1. .cluster-config.yaml (repo root)
2. ~/.config/ai4science-studio/cluster.yaml

If either exists, read it, show the user the current settings, and ask:

• Update — re-run discovery and update the file
• Keep — leave it as-is and exit

If neither exists, proceed to Step 1.

Step 1 — Auto-discover cluster environment

Run ALL of the following discovery commands silently (do not show raw output). Parse and store results for the questionnaire in Step 2.

# 1. Home directory (may not be /home/...)
echo "$HOME"

# 2. GPU architecture and count
rocminfo 2>/dev/null | grep -oP 'gfx\d+' | sort -u
rocm-smi --showproductname 2>/dev/null | head -20
ls /dev/dri/renderD* 2>/dev/null | wc -l

# 3. VRAM per GPU (MB → GB)
rocm-smi --showmeminfo vram 2>/dev/null | grep "Total" | head -1

# 4. ROCm version
cat /opt/rocm/.info/version 2>/dev/null || rocminfo 2>/dev/null | grep -oP 'HSA Runtime Version:\s*\K.*'

# 5. SLURM: available GPU partitions
sinfo -h -o "%P %l %G %D" 2>/dev/null | grep -i gpu

# 6. SLURM: user's accounts and associations
sacctmgr show associations where user=$USER format=account%30,partition%30,qos%30 -n 2>/dev/null

# 7. Container runtimes available
which apptainer 2>/dev/null && apptainer --version
which docker 2>/dev/null && docker --version 2>/dev/null
docker info 2>/dev/null | grep -i "amd\|runtime"

# 8. Common scratch / project directories (shared storage)
for d in /scratch/$USER /scratch /lustre /gpfs /tmp/$USER; do
    [[ -d "$d" ]] && echo "scratch: $d"
done

# 9. Node-local fast storage (for MIOpen cache, tmp writes during training)
for d in /scratch /local /nvme /tmp; do
    [[ -d "$d" && -w "$d" ]] && echo "scratch_local: $d" && break
done

# 10. RCCL / network interface discovery (for multi-node)
ip -o link show up 2>/dev/null | awk -F': ' '{print $2}' | grep -v lo
ibstat 2>/dev/null | grep "CA '" | awk -F"'" '{print $2}'

# 11. Internet connectivity from this node
curl -s --connect-timeout 5 -o /dev/null -w "%{http_code}" https://huggingface.co 2>/dev/null

# 12. Proxy settings
echo "HTTP_PROXY=${HTTP_PROXY:-<unset>}"
echo "HTTPS_PROXY=${HTTPS_PROXY:-<unset>}"

# 13. Existing SIF files
find "$HOME" /scratch /projects /opt -maxdepth 4 -name "*.sif" 2>/dev/null | head -20

# 14. Performance tooling (perf-analysis / perf-optimizer-loop recipes — optional)
# Look for an existing shared perf-tools dir (omnistat + TraceLens venv "perf-inspect/").
for d in /shared/*/tools /shared/perf-tools "$HOME/perf-tools"; do
    [[ -d "$d/perf-inspect" || -d "$d/omnistat-src" ]] && echo "perf_tools: $d"
done