cluster-operations

Skill: Cluster Operations

Expertise: Safe day-2 operations on self-hosted Kubernetes clusters — node drain, etcd ops, cert rotation.

When to load

When draining nodes for maintenance, rotating certificates, backing up etcd, or troubleshooting control plane issues.

Node Lifecycle Operations

# --- CORDON (stop scheduling new pods, don't evict existing) ---
kubectl cordon <node-name>
# Use case: pre-drain notification, temporary maintenance hold

# --- DRAIN (evict all pods, mark unschedulable) ---
kubectl drain <node-name> \
  --ignore-daemonsets \          # DaemonSet pods can't be evicted
  --delete-emptydir-data \       # required for pods using emptyDir
  --grace-period=60 \            # give pods time to shut down cleanly
  --timeout=300s                 # abort if takes > 5 minutes
# After drain: node is unschedulable and empty (except daemonsets)

# --- UNCORDON (return to service) ---
kubectl uncordon <node-name>

# --- Verify node is empty before maintenance ---
kubectl get pods -A --field-selector=spec.nodeName=<node-name>

etcd Backup (bare-metal / kubeadm)

# --- Take snapshot (run on a control plane node) ---
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key

# --- Verify snapshot ---
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-latest.db --write-out=table

# --- Restore snapshot (disaster recovery — only when cluster is down) ---
ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-latest.db \
  --data-dir=/var/lib/etcd-restored \
  --initial-cluster=master-1=https://192.168.1.10:2380 \
  --initial-advertise-peer-urls=https://192.168.1.10:2380 \
  --name=master-1
# Then update etcd static pod manifest to point to new data-dir

Certificate Rotation (kubeadm)

# --- Check certificate expiry ---
kubeadm certs check-expiration

# --- Renew all certificates (run on each control plane node) ---
kubeadm certs renew all

# --- Restart control plane components after renewal ---
# (kubeadm renews certs but doesn't restart static pods automatically)
for pod in kube-apiserver kube-controller-manager kube-scheduler; do
  kubectl -n kube-system delete pod -l component=$pod
done

# --- Update kubeconfig after cert renewal ---
cp /etc/kubernetes/admin.conf ~/.kube/config