kubernetes-operations

Kubernetes — pod debug decision tree

For ArgoCD-managed resources, also check the argocd-operations skill: direct mutations are reverted by selfHeal within ~3 minutes.

When to invoke

The pod's STATUS column tells you which branch to take. Always start with:

kubectl config current-context                  # confirm cluster/env BEFORE anything
kubectl describe pod <pod> -n <ns> | tail -40   # events at the bottom
kubectl logs <pod> -n <ns> [-c <container>] [--previous]
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -20

The Events: section at the end of describe is the single highest-signal source. Read it before anything else.

Pre-flight: is this resource Argo-managed?

Before any kubectl edit/patch/apply -f fix, check whether the resource is GitOps-owned:

kubectl get <kind> <name> -n <ns> -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}'
# managed-by indicators: argocd.argoproj.io/tracking-id, meta.helm.sh/release-name, app.kubernetes.io/managed-by

If managed: fix the source (chart/values/kustomization), not the cluster. See the argocd-operations skill.

Branch 1 — Pod never created (FailedCreate on the controller)

The pod doesn't exist yet; the ReplicaSet/StatefulSet/Job can't create it.

# Look at the controller's events, not the pod's (the pod isn't there)
kubectl describe rs <rs-name> -n <ns> | tail -30
kubectl describe statefulset <ss> -n <ns> | tail -30

Event message contains	Cause	Fix
forbidden: violates PodSecurity	Pod Security Admission rejecting the spec at the namespace's PSA level (restricted, baseline)	Either relax the namespace label pod-security.kubernetes.io/enforce or fix the pod spec (drop capabilities, runAsNonRoot, etc.)
exceeded quota / forbidden: exceeded quota	ResourceQuota in the namespace	kubectl describe resourcequota -n <ns> to see what's exhausted
admission webhook "..." denied the request	A ValidatingWebhookConfiguration rejected the pod	kubectl get validatingwebhookconfigurations, inspect the named webhook's policy; the webhook's controller logs explain why
serviceaccount "X" not found	SA referenced in pod spec doesn't exist in this namespace	Create the SA, or fix the spec; common with Helm chart values mismatch
persistentvolumeclaim "X" not found (StatefulSet)	The PVC template name doesn't match what was provisioned, or volumeClaimTemplate changed	StatefulSet PVCs are immutable; delete and recreate, or revert the template
error looking up service account ... no token	ServiceAccount exists but no token Secret (IRSA / GKE WI setups)	Check Workload Identity bindings (GCP) or service-account token projection