gke-ai-troubleshooting-tpu-vbar-oom

TPU Connection Failure and VBAR OOM Troubleshooting

Use this skill to systematically diagnose and prevent vbar_control_agent segfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.

⚠️ Prerequisites

• Cloud Logging must be enabled for the project.
• Access to the project and cluster via gcloud or equivalent tool.

🔍 Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Independently gather required context using available GCP/GKE tools or use the provided {variable} placeholders:

• {project_id}: The GCP Project ID (e.g., customer-ai-project-123).
• {cluster_name}: The GKE Cluster Name (e.g., tpu-cluster-prod).
• {node_name}: The Node Name or Instance ID (e.g., tpu-node-1).
• {workload_name}: The Workload Name / JobSet Name (e.g., my-training-job-456).
• {namespace}: The Workload Namespace.
• {issue_time}: The timestamp of the issue (e.g., 2026-04-14T20:00:00Z).

Time Handling & Execution Rules

1. Window Calculation: If an issue timestamp {issue_time} is provided, calculate the query time window as [{issue_time} - 30m] to [{issue_time} + 30m].
   • Let {start_time} = {issue_time} - 30m
   • Let {end_time} = {issue_time} + 30m
2. Informational vs. Live Execution: If the user request is informational or query-formulation (e.g. "How can I check...", "How do I determine..."), or if live GCP project resources are not actively targetable, directly output the calculated time window, log names, and Cloud Logging filter templates without attempting live log execution commands.

Step 1: Check for vbar_control_agent OOMs

Look for specific out of memory messages from vbar_control_agent in serial console logs (serialconsole.googleapis.com%2fserial_port_1_output).

• Tool to use: query_logs (for live diagnostics)
• Filter Templates:

Serial Console Logs (OOMs):

logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output"
AND labels."compute.googleapis.com/resource_name"="{node_name}"
AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"