gke-reliability

GKE Reliability Skill

This skill provides workflows for configuring your GKE cluster and workloads for high availability and reliability.

Workflows

1. Verify Cluster High Availability

Check if the cluster is regional or has multi-zonal node pools.

Command:

gcloud container clusters describe <cluster-name> --region <region> --format="json(location, locations)"

If location is a region (e.g., us-central1), the control plane is regional. If locations has multiple entries, nodes are spread across multiple zones.

2. Configure Pod Disruption Budgets (PDB)

PDBs ensure that a minimum number of pods are available during voluntary disruptions (like node upgrades).

Check existing PDBs:

kubectl get pdb -n <namespace>

Example Manifest:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

3. Configure Health Probes

Ensure all production containers have Liveness, Readiness, and optionally Startup probes.

• Readiness Probe: Determines when a container is ready to start accepting traffic.
• Liveness Probe: Determines when to restart a container.
• Startup Probe: Disables liveness and readiness checks until the app has started up.

Check workload probes:

kubectl get deployment <app-name> -n <namespace> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"

4. Graceful Shutdown

Ensure applications handle SIGTERM signals gracefully and have an appropriate terminationGracePeriodSeconds set (default is 30s).

5. Topology Spread Constraints

Ensure pods are spread across zones or nodes to avoid correlated failures.

Example Manifest excerpt:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule # or ScheduleAnyway
      labelSelector:
        matchLabels:
          app: my-app