together-volcano

Volcano on Together GPU clusters

Volcano is a Kubernetes-native batch scheduler. Its key property is gang scheduling: a group of pods is placed all-at-once or not at all, so distributed training never starts with only some workers running. Use it when a job needs N pods to run together or not at all.

Public cookbook: https://docs.together.ai/docs/volcano-on-gpu-clusters. Pair this skill with the together-gpu-clusters skill, which covers creating the cluster and configuring kubectl.

Preconditions

• A Together Kubernetes GPU cluster in the Ready state, with kubectl pointed at it (tg beta clusters get-credentials <cluster_id> --set-default-context).
• GPU nodes expose nvidia.com/gpu; the NVIDIA device plugin is preinstalled on Together clusters. Confirm with:

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'

Install

Pin the version. Do not use master.

kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.15.0/installer/volcano-development.yaml
kubectl -n volcano-system wait --for=condition=Available deployment --all --timeout=180s

A default queue is created automatically (kubectl get queue). Helm is an alternative: helm repo add volcano-sh https://volcano-sh.github.io/helm-charts && helm install volcano volcano-sh/volcano -n volcano-system --create-namespace.

Create a queue

A queue caps and weights the resources a set of jobs can use.

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: research
spec:
  reclaimable: true          # give idle capacity back to other queues
  weight: 1                  # relative share under contention
  capability:                # hard ceiling for this queue
    nvidia.com/gpu: 8

Attach shared storage (optional)

Together provisions a static PersistentVolume named after the cluster's shared volume. Bind a ReadWriteMany PVC to it so every worker shares datasets and checkpoints, then mount it in the job (below).