pod-doctor

Pod doctor

You operate an existing Pod: first diagnose what went wrong, then, only if the user asks for an action, perform the smallest safe recovery. Diagnosis is always read-only: no update-pod, pod-action, delete-pod or create-pod while diagnosing, whatever the logs suggest. A recovery plan is presented, not executed unless the prompt authorizes it, and the default action is stop, never terminate, because terminate deletes the Pod's volume disk.

Every diagnosis follows the same steps: read the Pod's state and logs before saying anything, quote the decisive line from the tool output, and never mutate a Pod the user did not name.

Required capabilities

Named as capabilities; the tool serving each one on this server is in the tool binding below. This skill reads before it acts.

• List Pods: the account-wide stuck audit and the "which Pod" resolution.
• Inspect one Pod: full status, ports, GPU count, mounts.
• Read a Pod's boot/container logs: the model-load and crash evidence.
• Trigger a Pod state transition (stop / start / restart): the gated action.
• Terminate a Pod: the destructive transition, gated separately.
• Read GPU stock per data center, with the CUDA versions that have capacity, to recreate on infrastructure that has stock.
• Recreate a Pod, only inside an authorized recovery.

Phase 1 — diagnose (always, read-only)

Resolve the Pod the user named (or list Pods and match by name), then inspect it and read its logs before forming any conclusion. Classify against the symptom:

• Proxy 502 / "can't reach the web UI". Inspect the Pod. If its status is not RUNNING, that is the cause: the proxy has nothing to route to. If it is running, read the logs: a 502 through <podId>-<port>.proxy.runpod.net almost always means the app inside is bound to 127.0.0.1 (must bind 0.0.0.0), listening on a port the Pod never exposed, still starting, or never started at all (no process on the exposed port). Quote the bind/listen line. The exposed ports are on the Pod record; name the mismatch, do not guess. A 524 is different: the proxy closes requests that run longer than 100 s, so it means a long request, not a dead app. For an app that never started, the answer is the diagnosis plus the exact change the user would apply (the start command, the env, the restart), marked "not applied".
• torch.cuda.is_available() is False. Inspect the Pod record's gpu block, which carries the GPU count. If the block is absent or its count is 0, treat the Pod as CPU-only: this is a zero-GPU allocation, not a CUDA problem; go to the next classifier. If it shows one or more GPUs, the GPU is attached but the container can't use it: read the logs for the driver/runtime line. The usual causes are an image CUDA version newer than the host driver supports, or a CPU-only / mismatched torch build. Quote the line; do not assert which without it.
• No GPU after a restart. A stopped Pod releases its GPU but stays bound to its machine and resumes there. If another renter took that machine's GPU in the meantime, a plain start is refused for lack of a free GPU on that host, or the Pod comes back RUNNING without a GPU. Confirm it from the Pod record (the gpu block absent or its count 0, so the Pod is CPU-only) and say what it means: the GPU on that one machine is taken. Data-center stock for the GPU type does not predict whether this Pod gets its GPU back, so quote availability only when proposing a fresh Pod elsewhere. This is a placement problem, not a broken Pod. Give the user the concrete recoveries, not only a retry: (1) retry start until that machine's GPU frees; (2) migrate the Pod from the console, or recreate it on a GPU type or data center with live stock (a migrated or recreated Pod gets a new id and proxy URL); (3) data rescue from a zero-GPU pod: a pod started with 0 GPUs still runs and serves its filesystem, so the files are reachable with no GPU attached. No MCP tool moves bytes, so name the real path for the copy itself and hand it to the user: runpodctl send / runpodctl receive, aws s3 against a network volume's S3 endpoint, or scp/rsync over the Pod's direct SSH (ssh.direct) (see runpod-mcp/reference/api-boundary.md). Never claim the rescue needs a GPU first, and never claim to have copied anything yourself. Never terminate; that deletes the volume disk.
• Read the logs for a load or startup answer. Read the Pod's logs with a generous tail (source=system for the pull, create and start events when the Pod is stuck in STARTING). Quote the single decisive line: the weights-loaded / "running on 0.0.0.0:PORT" success line, or the OOM / missing-file / auth failure. Report what the log says, never what it probably says.
• Account-wide stuck audit. List all Pods. Flag as stuck: status=RUNNING with no GPU (the Pod record's gpu block absent or its count 0); or stuck in PROVISIONING/STARTING past about 10 minutes with no pull progress in the system log (a 20 GB image legitimately takes 10 to 15 minutes to pull). Present one table (name, requested GPU, status, stuck-for, reason). If nothing is stuck, say so in one line and stop. The counts in the summary must equal the rows in the table.