k8s-debug

Kubernetes Debugging Skill

Mindset

Mental Model: Kubernetes debugging follows a layered approach—start broad (cluster health), narrow to affected components (pods, services), then drill into specific failures (logs, events, resource constraints). A pod's status is always a summary, never the diagnosis — CrashLoopBackOff and ImagePullBackOff name the symptom, not the root cause, and you must read events and logs before acting on the label alone.

Decision Framework:

1. Gather context before jumping to solutions. Check kubectl get events and describe resources first — this step is mandatory, never optional, even when the fix seems obvious.
2. Verify assumptions about selectors, labels, and namespaces—label mismatches are the most common root cause and must never be assumed away.
3. Test hypotheses systematically: network → resource → configuration → application.
4. Document findings as you go—Kubernetes issues often involve multiple interacting failures, and an undocumented fix is one nobody else can verify. A common pitfall: treating a restart as a fix when it only cleared a transient symptom — the same gotcha resurfaces on the next scheduling cycle if the root cause was never found.

When to use this skill:

• Pods stuck in Pending, CrashLoopBackOff, ImagePullBackOff, or Error states
• Services not routing traffic despite healthy pods
• Resource exhaustion (OOMKilled, CPU throttling)
• Deployments failing to roll out or stuck in progress
• Network policies blocking expected traffic

When NOT to use this skill:

• The failure is already fully explained by a known recent change (a deploy minutes ago introduced a bad config) — roll back directly rather than running the full diagnostic funnel.
• The question is capacity planning (steady-state resource sizing, autoscaler tuning) rather than an active failure — that is a design decision, not a debugging session.
• You lack read access to the affected namespace — request access first; guessing from an adjacent namespace produces false leads.