gke-stockout-investigator

GKE Stockout Diagnosis & GitOps Remediation Skill

This skill guides the Platform Agent on how to handle, diagnose, and remediate GKE capacity stockout alerts (scale.up.error.out.of.resources or equivalent) received via Google Cloud Pub/Sub.

Workflow

1. Notify User of Investigation Start (MANDATORY FIRST STEP)

Before executing ANY terminal commands, scripts, or diagnostics, you MUST immediately call the send_notification tool (or mcp_platform_control_send_notification) to announce that the stockout alert was received and investigation is starting:

send_notification(message="🚨 GKE Stockout Investigation Started\nWorkload: <workload_name>\nCluster: <cluster_name>\nDetails: A GKE capacity stockout alert is confirmed. I am starting an investigation and diagnosis.")

CRITICAL: You MUST invoke send_notification (or mcp_platform_control_send_notification) as an actual tool call. Do NOT output this notification as plain markdown text. After calling the tool, proceed immediately to Step 2 in your next tool turn without stopping.

[!IMPORTANT] When running in a background or PubSub context, NEVER use the execute_code tool or write Python scripts/subshells, as they trigger command approval safeguards and block indefinitely waiting for human approval. Only execute standard command-line tools directly (kubectl, gh, gcloud) or dedicated tools like send_notification.

2. Pre-Diagnosis Verifications (Duplicate PRs & False Signal Checks)

After sending the initial notification, perform two critical safety checks to see if you should stop immediately:

A. Check for Existing Relevant Pull Requests (Duplicate Prevention)

To prevent duplicate effort and redundant PRs, inspect currently open Pull Requests in the repository:

1. Resolve the GitOps repository as <owner>/<repo>, and pass it as --repo on every call from here on. The pod is not a working copy (Step 3 explains why), so there is no local repository for a command to infer one from, and a call that omits it fails on repository resolution rather than returning a list — which is the duplicate check silently never happening. Query the $GITOPS_STATE_CONFIGMAP ConfigMap: