volcano-gang-scheduling

Gang Scheduling Diagnosis

This is a diagnostic guide for Gang scheduling issues in Volcano. Gang scheduling requires that all members of a PodGroup be scheduled simultaneously. If the cluster cannot satisfy the minMember requirement, none of the pods will be scheduled.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify PodGroups or resource configurations.

When to Use This Guide

Use this skill when:

• PodGroup status is Inqueue but member Pods remain Pending
• Events contain minMember related errors
• Volcano Job has minAvailable or minMember that cannot be satisfied
• Some member Pods are running, others are Pending, and the entire group won't start
• You see FailedScheduling events mentioning Gang constraints

Understanding Gang Scheduling

Gang scheduling in Volcano ensures that either all members of a workload are scheduled, or none are. This is crucial for distributed workloads like MPI, TensorFlow, PyTorch where partial scheduling is wasteful.

Key Concepts:

• minMember (in PodGroup spec): Minimum number of pods that must be scheduled simultaneously
• minResources (in PodGroup spec): Aggregate resource floor (e.g., total GPUs) that must be available — both minMember and minResources must be satisfied if set
• minAvailable (in Job spec): Similar concept at Job level
• The scheduler checks if there are simultaneous resources for all minMember pods before allocating

Diagnostic Steps

Step 1: Identify the PodGroup

Find the PodGroup associated with the pending pods:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.annotations.scheduling.volcano.sh/pod-group}'

Step 2: Check PodGroup Status

Get detailed PodGroup information:

kubectl get podgroup <podgroup-name> -n <namespace> -o yaml

Key fields to examine:

Field	Meaning	What to Look For
spec.minMember	Minimum pods required	Is this number achievable?
spec.minResources	Aggregate resource floor	Is total cluster capacity sufficient?
status.phase	Current scheduling phase	Should be Inqueue for ready-to-schedule
status.running	Currently running pods	Compare to minMember
status.pending	Pending pods	These are waiting for Gang constraint
spec.queue	Queue name	Check if queue has sufficient resources