aiml-gpu-training-cluster-investigation

GPU Cluster Evidence, Readiness, and Fault Verdicts

For GPU clusters on SageMaker HyperPod (Slurm or EKS), AWS ParallelCluster, and self-managed EC2 or EKS. Generic investigation finds the obvious signals; this skill adds what it gets wrong: silent evidence, wrong-headline hardware verdicts, and predictable failures before a long run. Read-only. Never reboot, replace, update, or delete anything, and never read training data, checkpoints, or model weights.

Critical rules R1 to R11 (apply in every mode, in this order)

R1. Answer in one pass, and always leave room to answer. In chat, do not stop to ask a question and do not hand off to a separate investigation before answering. If an input is missing, use the default (impact window: last 24 hours; run length: evaluate 24, 48, and 72 hours), state the assumption, and mark dependent checks Needs input. For "slow" or performance questions with no time given, use the last 72 hours. Offer follow-ups only after the answer. Budget the evidence gathering so the answer always gets written. An investigation that runs out of room before it reports is worth nothing to the operator, and it is worse than a partial answer because it looks like a failure rather than a finding. So: collect the mandatory evidence for the mode first (R2, R4, R5, and for Mode P the P1 to P6 core), then write the report. Pick up the optional checks only with what is left. If you notice you are deep into tool calls and have not yet produced an answer, stop collecting and report what you have, marking everything unreached as Not checked with the call that would close it. Never end a turn with evidence gathered and no verdict. R2. Inventory and capability profile. HyperPod: sagemaker.DescribeCluster and ListClusterNodes (paginate). Read NodeProvisioningMode from DescribeCluster: if it is Continuous, also pull sagemaker.ListClusterEvents for the window (see rule R11), which is the only timeline source that survives broken log delivery. On any other value the call is unsupported and must be skipped, not retried. EC2/ParallelCluster/EKS: ec2.DescribeInstances. For every GPU instance type: ec2.DescribeInstanceTypes (strip HyperPod ml.): GPU count, EfaSupported, MaximumEfaInterfaces; per EC2 node, attached efa/efa-only interfaces. Report <attached> of <max>, where attached = interfaces with InterfaceType efa or efa-only (the primary ENA interface does not count unless it is efa). HyperPod nodes are not visible to DescribeInstances: say so. On NVSwitch types, search every log source found under rule R4 for Started "Nvidia Fabric Manager" before saying it is not confirmed. R3. Node identity survives replacement. A HyperPod reboot keeps the instance ID; a replace gives the node a new instance ID in the same instance group, so the current ID will never appear in the replace request. Query CloudTrail by event name, not by instance ID: cloudtrail.LookupEvents with LookupAttributes=[{AttributeKey: EventName, AttributeValue: BatchReplaceClusterNodes}], then again for BatchRebootClusterNodes, BatchDeleteClusterNodes, and UpdateCluster, with StartTime = window start minus 6 hours and EndTime = now as full ISO-8601 UTC timestamps, paginating with NextToken. Keep events whose requestParameters.clusterName is this cluster. A nodeIds entry that is not in the current ListClusterNodes output was replaced; the instance group whose node has a LaunchTime just after that event is the replaced group. That operator or automatic call is the explanation for the node going Pending (Branch E), not hardware. R4. Find every log source by substring, not prefix. Call logs.DescribeLogGroups with logGroupNamePattern (case-sensitive substring) = the cluster name, then again for kernel, messages, syslog, journal, and gpu, paginating with nextToken. Never search only /aws/parallelcluster or /aws/sagemaker prefixes: customer pipelines use other names (for example /aws/<pipeline>/<cluster>/kernel). Evaluate every source found. R5. Prove coverage before any "no errors". For each node and source: find the stream that carries kernel: lines, then bin that exact stream by hour across the window padded by one hour. Always name the evidence you used: quote the full log group name and the exact log stream name for every node in the coverage table, and again in the answer text. A coverage claim without the group and stream it rests on is not auditable, so the operator cannot re-run it. Any empty hour means Not observable for that hour. First and last event times are not proof, and the time of the last kernel: line is not when logging stopped: a healthy kernel goes quiet. Liveness comes only from the hourly bins of all lines in that stream. A node is Measured if one source passes. HyperPod: a missing SagemakerHealthMonitoringAgent/<group>/<instance-id> stream means No HMA detections when the cluster log group is otherwise live. NCCL transport with no NCCL INFO lines anywhere is Not observable; never infer it from the instance type. R5a. Name every resource you looked at, by ID. A finding the operator cannot re-run is not a finding. Whatever you analysed, put its identifier in the answer: the FSx file system (fs-...) behind any storage claim, the instance IDs (i-...) behind any node claim, the cluster name, the capacity reservation (cr-...) behind any capacity claim, and the log group and stream behind any log claim as R5 already requires. "The file system was saturated" or "the metrics looked fine" names nothing and cannot be checked. This applies to the resource you cleared as much as the one you blamed, since ruling something out is only useful if the reader knows what was ruled out. R6. Verdict per node, headline to match. REPLACE, REBOOT, LEAVE ALONE, MONITOR, or NOT OBSERVABLE, against the evidence bar in references/incident-branches.md (Step 4b). Application-class Xids (for example 13, 31) or HMA reason: XidUserAppError with the node Running is LEAVE ALONE. Never headline "hardware error" unless the verdict is REPLACE or REBOOT on hardware grounds. R7. Label every cause Proven (measured signal on the affected node, before the failure, nothing competing) or Hypothesis (to validate) with the one confirming measurement. A spike at the same time is correlation. FSx without a saturated metric is not a proven cause. Only a Proven cause may be called the root cause, in the headline or in a branch table. Otherwise write Leading hypothesis: <cause>, or Root cause: Not observable when the deciding evidence is missing (for example a dead control-plane log). Never write "Proven mechanism" for something whose trigger or removal path you did not observe. Utilization metrics from FSx (NetworkThroughputUtilization, DiskIopsUtilization, and similar) and GPUPowerUtilization are already percent from 0 to 100: a value of 0.9 is 0.9 percent. Quote the raw value with a percent sign. R8. Recovery questions always state three things: whether automatic node recovery is on (NodeRecovery), what it does (reboot or replace the node), and that the job resumes only with checkpoints plus the orchestrator's auto-resume (Slurm on HyperPod: srun --auto-resume=1). R9. Capacity Blocks begin terminating instances 30 minutes before the end time (60 for UltraServers); blocks end at 11:30 UTC and termination starts at 11:00 UTC on the last day. For a planned run, write out: usable until = end time minus the lead time; run end = start plus run length; hours covered = usable until minus start. Give every value as a full UTC date and time, and check the latest safe start is not already in the past. R10. Rule out the frequent non-GPU causes in references/cluster-edge-cases.md before blaming hardware: subnet IP or network interface exhaustion, ParallelCluster bootstrap failures and protected mode, EFA nodes in a public subnet, a Capacity Block not yet active, and the FSx maintenance window. HyperPod does not export system metrics to CloudWatch, so HyperPod GPU activity is Not observable there. R11. When the logs are dead, ask the control plane. On a HyperPod cluster with NodeProvisioningMode = Continuous, sagemaker.ListClusterEvents gives you a node and cluster timeline that owes nothing to a log agent, so it keeps answering when a stream has gone silent or a node has disappeared. Filter the window with EventTimeAfter and EventTimeBefore, narrow with NodeId or InstanceGroupName, sort with SortBy=EventTime, and page through NextToken. Where a Description is not self-explanatory, DescribeClusterEvent has the detail. Note that the response has no severity or level field at all, so any grouping you apply is your own and should be described that way. If NodeProvisioningMode is anything other than Continuous the call is not supported; write ListClusterEvents not supported in the coverage table and carry on. What you must not do is report a dead log as "no events" without either trying this source or saying it was unavailable.