chaos-engineering

Skill: Chaos Engineering

Expertise: LitmusChaos experiments, manual K8s chaos, network partition testing, graceful degradation validation.

When to load

When designing chaos experiments, validating failover behavior, verifying SLO headroom, or onboarding a service to chaos testing.

Chaos Experiment Design Principles

1. Define steady state first
   → What does "working" look like? (SLI baseline: error rate < 0.1%, p99 < 200ms)

2. Hypothesize
   → "If 1/3 of pods die, the service will continue serving with p99 < 500ms"

3. Blast radius control
   → Start with staging. Start with 1 pod. Increase gradually.

4. Abort conditions
   → Auto-stop if error rate > 1% or p99 > 1s for > 2 min

5. Document and act
   → Passed = evidence of resilience. Failed = fix + re-test. Never just accept failure.

Manual Chaos (no tooling needed)

# ── Pod kill (test restart recovery) ──────────────────────────
kubectl delete pod <pod-name> -n production
# Watch: kubectl get pods -n production -l app=my-service -w
# Expected: new pod starts, readiness probe passes, 0 user-visible errors

# ── Kill all pods in deployment (test rolling restart recovery) ──
kubectl rollout restart deployment/my-service -n production
# Watch error rate during rollout

# ── Simulate OOMKill ──────────────────────────────────────────
kubectl exec -it <pod> -n production -- sh -c \
  "dd if=/dev/zero of=/dev/shm/blob bs=1M count=600"
# Expected: pod OOMKilled, restarted, alert fired, no user impact

# ── Resource pressure on node ─────────────────────────────────
kubectl run stress --image=polinux/stress --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"worker-01"}}}' \
  -- stress --cpu 4 --vm 1 --vm-bytes 2G --timeout 120s

# ── Network partition: isolate a pod (Cilium + network policy) ──
# Apply a policy that drops all traffic from/to the pod
kubectl apply -f - << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: chaos-isolate, namespace: production }
spec:
  podSelector: { matchLabels: { chaos-target: "true" } }
  policyTypes: [Ingress, Egress]
EOF
kubectl label pod <pod> chaos-target=true -n production
# Observe: circuit breakers trip, retries, fallback behavior
# Cleanup:
kubectl delete networkpolicy chaos-isolate -n production
kubectl label pod <pod> chaos-target- -n production