kubernetes-operations

Kubernetes Operations

Kubernetes does not execute your intent once — it continuously reconciles the cluster's actual state toward your declared state, forever. Every operational problem is really a question about that loop: what did you declare, what does the controller see, and where does the gap live. Most outages trace back to a pod spec that never told the scheduler or kubelet what it actually needed.

The controller can only be as good as the information you give it — requests, limits, and probes are not bureaucracy, they are the interface you use to talk to the control loop. Debug by reading what the control loop already told you, not by guessing.

For a command-by-command debugging playbook and a table of pod failure signatures, read references/kubectl-debugging.md.

1. Set requests as a promise, limits as a leash

Requests are what the scheduler uses to place a pod — they must reflect real steady-state usage or you get either wasted nodes or overcommitted ones. Limits cap the ceiling and, for memory, define the OOM-kill boundary; for CPU they cause throttling, not killing. Never set a memory limit without a request, and never guess both from nothing — profile first.

• CPU limits throttle, they don't fix: a CPU limit that's too tight shows up as latency, not crashes, so check kubectl top and cgroup throttling stats before assuming code is slow.
• Memory limits kill: exceeding a memory limit is an immediate SIGKILL with reason OOMKilled, visible in kubectl describe pod.
• QoS class follows from this: requests == limits gets Guaranteed, which matters for eviction order under node pressure.

Done when: every container has a memory limit backed by observed usage, and CPU is only limited if you've confirmed throttling is an acceptable tradeoff.

2. Separate what "alive" means from what "ready" means

Liveness, readiness, and startup probes answer three different questions and conflating them causes the most common self-inflicted outage: a slow-starting app gets liveness-killed in a crash loop because there's no startup probe to give it room, or a readiness probe that's really a liveness check pulls healthy pods out of service during a blip.