gke-alert-configuration

GKE Alert Configuration

This skill provides guidelines and best practices for creating robust, high-signal alerting policies for Google Kubernetes Engine workloads using Google Cloud Managed Service for Prometheus and Terraform. It ensures comprehensive coverage of the 4 Golden Signals and key cluster health metrics while minimizing alert noise.

Critical Rules

• Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes:
  • This skill is strictly scoped to Google Kubernetes Engine (GKE) workloads, clusters, and services using PromQL and Google Cloud Managed Service for Prometheus.
  • Do not use for non-GKE compute runtimes, such as standalone Compute Engine virtual machines or standalone Cloud Run services without GKE.
  • STOP AND RESPOND DIRECTLY (Do Not Edit Files): When the user requests alert configuration for non-GKE compute infrastructure:
    1. Do not write, create, edit, or validate any Terraform files on disk.
    2. Immediately stop and respond directly to the user in chat:
       • Explicitly Clarify Out-of-Scope: State clearly that standalone Compute Engine virtual machine monitoring or standalone Cloud Run monitoring is out of scope for this GKE-specific PromQL alerting skill, which is designed specifically for GKE workloads using Google Cloud Managed Service for Prometheus and PromQL.
       • Do Not Generate GKE PromQL Alerts: Do not create or generate Kubernetes PromQL alert policies or fabricate Kubernetes container, pod, or node resources for non-GKE infrastructure.
       • Redirect the User: Guide and redirect the user to standard Google Cloud Monitoring metrics, such as compute.googleapis.com/instance/cpu/utilization or run.googleapis.com/request_latencies, using standard google_monitoring_alert_policy with condition_threshold or MQL, or recommend the relevant specialized Cloud observability skill.
• Mandatory kube-state-metrics (KSM) Cost Guardrail:
  • Deploying open-source kube-state-metrics in Google Cloud Managed Service for Prometheus incurs billable metric ingestion costs.
  • STOP AND ASK PERMISSION FIRST (Do Not Edit Files): When a requested alert rule relies on Tier 2 KSM metrics (such as kube_cronjob_*, kube_pod_status_phase, kube_persistentvolume_*, kube_deployment_*, kube_statefulset_*, kube_job_*, or kube_daemonset_*), do not write, create, edit, or validate any Terraform files or generate alert policies before obtaining user approval.
  • Instead, you must immediately stop and respond directly to the user to:
    1. Alert the user that the requested alert requires kube-state-metrics.
    2. Explain the cost impact: Detail that kube-state-metrics incurs billable sample ingestion costs in Google Cloud Managed Service for Prometheus.
    3. Ask for explicit permission: Ask the user for explicit permission before assuming, enabling, or generating KSM-dependent alert configurations.
    4. Recommend filtering or allowlisting: Suggest and recommend filtering or allowlisting only the specific required metrics, such as using a PodMonitoring resource with metricRelabeling (action: keep) or KSM --metric-allowlist to minimize ingestion costs. Provide a concrete allowlist example.
  • Always prefer Non-KSM Native Alternatives (Tier 1 cAdvisor or native GKE metrics documented in metrics_and_alerts_catalog.md) whenever possible, such as using container_memory_working_set_bytes and container_spec_memory_limit_bytes instead of kube_pod_container_resource_limits.
  • Explicit Tier and Cost Surcharge Identification in Response: In every response where you generate or recommend an alerting policy, you must explicitly state its classification tier and cost impact:
    • Tier 1 native or standard metric (GKE built-in metrics, cAdvisor container_*, kubelet volume stats, native kubernetes.io/node/status_condition, and control-plane metrics. Every kube_* series the references name (kube_node_status_condition, kube_node_spec_taint and kube_pod_container_status_restarts_total among them) is a kube-state-metrics series and therefore Tier 2, whatever the KSM-required or tier columns in the references say; see metrics_and_alerts_catalog.md): State that it is a Tier 1 native or standard metric with zero KSM cost surcharge.
    • Tier 2 KSM metric: State that it is a Tier 2 KSM-dependent metric and follow the permission and allowlisting guardrail above. (Tip: Generally, metrics with the kube_ prefix that represent resource state or metadata belong to Tier 2).
• Plan-Validate-Execute Loop for Approved File Edits: When modifying, adding, or merging approved Terraform files on disk in a workspace, follow the three-phase workflow:
  1. Plan: Draft a structured change plan (changes.json) containing proposed policy resource names, PromQL expressions, grouping labels, and durations.
  2. Validate: Run the pre-edit validation script (python3 "$HERMES_HOME"/skills/gke-alert-configuration/scripts/validate_config.py --plan changes.json) to check bracket balance, time-window and offset formats, duration rules, and duplicate signals. The duplicate check compares only entries that carry a signal_type (for example saturation_memory), so give each planned policy one. Write the plan as {"policies": [{"resource_name": ..., "query": ..., "duration": ..., "signal_type": ...}]}: the script reads a duration only from the duration key and takes a missing one as 0s, so a duration under any other key passes the duration rule unchecked. It does not parse PromQL, so a pass does not prove the query is valid.
  3. Execute: After the plan passes validation, apply or merge changes in-place into the target Terraform configuration (alerts.tf).
  4. Note: When answering questions or providing Terraform snippets directly in chat where no disk modification is requested, output the complete, valid Terraform HCL block in your response.
• Configure the 4 Golden Signals and Cluster Health: Always ensure the target Kubernetes workload or service has the following alerting coverage:
  1. Latency (P95 response time)
  2. Errors (Multi-Window Multi-Burn-Rate SLO alerts, such as Fast Burn 1 hour / 5 minutes with factor 14.4, Slow Burn 6 hours / 30 minutes with factor 6.0; do not use simple static ratios)
  3. Traffic (Sudden drop or complete metric disappearance using absent() or default 0 syntax, or overload spikes)
  4. Saturation (Memory Limit Utilization Only): When describing or configuring alert policies for a cluster or project, include ONLY Memory Saturation (container_memory_working_set_bytes / container_spec_memory_limit_bytes). Do NOT include CPU saturation alerts or list container_cpu_usage_seconds_total as an alert metric because CPU is compressible and throttled by CFS quotas rather than causing uncompressible fatal termination (OOM).
  5. Cluster Health (Pod CrashLooping, Node NotReady)
• PromQL Only (Managed Prometheus): You must use condition_prometheus_query_language with PromQL. Do NOT use MQL or standard condition_threshold unless explicitly requested. Google Cloud Managed Service for Prometheus is the standard telemetry ingestion path for GKE.
• Terraform Only: Write the generated observability configuration ONLY as Terraform (.tf) files, such as alerts.tf and variables.tf.
• Dynamic Multi-Resource Alerting (No Hardcoding): You must not hardcode specific pod names, node names, or service names in alerting conditions unless explicitly requested. Alerting policies must be written to cover resources dynamically:
  • Always use grouping aggregations (by (cluster, namespace, service, pod, container)) instead of filtering to a single instance. This allows a single alert policy to dynamically track each service or pod separately.
  • Always declare and use Terraform variables for project_id, cluster_name, and namespace (var.project_id, var.cluster_name, var.namespace) to make the configuration reusable across environments. Always define these variables in variables.tf (or within the configuration) and reference all three in policies or PromQL label matchers.
• No Redundant Duration Windows on Lookbacks:
  • When PromQL expressions already use an aggregated lookback window (such as increase(...[15m]) > 3 or multi-window SLO burn rates), the query time window already smooths out transient spikes.
  • Adding a Terraform duration on top of a PromQL lookback window increases the Mean Time to Detect (MTTD) without providing additional smoothing benefits.
  • In these cases, set Terraform duration = "0s" (or "60s"). Do not enforce duration = "300s" on top of [15m], which delays critical crashloop alerts by up to 20 minutes total (15 minutes + 5 minutes).
  • Use duration = "300s" only on instantaneous gauge conditions, such as kube_node_status_condition == 0.
• Use SLO Burn Rates Instead of Simple Ratios: For error rate alerting, always generate Multi-Window Multi-Burn-Rate (MWMBR) SLO alerts (such as 14.4x burn rate over 1 hour and 5 minute windows for a 99% SLO) rather than simple error rate ratios (rate(5xx)/rate(total) > 0.05), which produce excessive false alarms on low traffic.
• Robust Traffic Drop Detection (absent() / default 0): When monitoring for traffic drops to zero, do not use rate(...) == 0 alone because Prometheus time series disappear completely when no requests occur (evaluating to an empty vector rather than 0). Use default 0 syntax, such as sum(rate(...[5m])) default 0 == 0, or absent(...) == 1.
• Notification Channels: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel, configure the alerts to use it. Otherwise, you must prompt the user in your response to ask if they would like to configure one.
• Consult GKE Metrics and Open-Source Alerts Catalog: When designing or generating evaluation suites or alerting policies, consult metrics_and_alerts_catalog.md for public GKE metrics (kubernetes.io/) and open-source Kubernetes alerts (awesome-prometheus-alerts).
• Plain English Response: You must include a plain English explanation for what the alerts do in your response. Explain what the alert measures, what the threshold represents, and what a trigger indicates.
• User Labels: Include a user_labels block in all google_monitoring_alert_policy resources to track policies created by this skill: