capacity-planning

Skill: Capacity Planning

Expertise: Traffic forecasting, per-pod resource modeling, node pool sizing, cluster capacity headroom, VPA/HPA tuning for growth.

When to load

When planning for growth, validating current cluster headroom, sizing node pools, or preparing for a high-traffic event (sale, launch).

Traffic Forecasting

# Current RPS baseline (7-day average)
avg_over_time(
  sum(rate(http_requests_total{service="checkout-service"}[5m]))[7d:5m]
)

# Peak RPS (7-day p99)
quantile_over_time(0.99,
  sum(rate(http_requests_total{service="checkout-service"}[5m]))[7d:5m]
)

# Week-over-week growth rate
(
  avg_over_time(sum(rate(http_requests_total[5m]))[7d:5m])
  /
  avg_over_time(sum(rate(http_requests_total[5m]))[7d:5m] offset 7d)
) - 1
# e.g. 0.08 = 8% weekly growth → ~3.5× in 6 months

Per-Pod Resource Modeling

Model: what resources does 1 pod consume per RPS unit?

Step 1: current pod metrics
  - pods = 4 (HPA current)
  - RPS = 200 req/s (avg)
  - CPU per pod = 320m (avg), 480m (p99)
  - Memory per pod = 280Mi (avg), 380Mi (peak)

Step 2: per-RPS resource cost
  - CPU per RPS = 320m / (200/4) = 6.4m CPU per RPS
  - Mem per RPS = 280Mi / (200/4) = 5.6Mi per RPS

Step 3: future requirements at 2× traffic (400 RPS)
  - CPU needed = 400 × 6.4m = 2560m = 2.56 cores
  - Mem needed = 400 × 5.6Mi = 2240Mi ≈ 2.2Gi
  - Pods needed (at 70% CPU target) = 2560m / (500m × 0.7) = 7.3 → 8 pods min
  - Update HPA maxReplicas to accommodate

Cluster Capacity Check

# Total cluster allocatable resources
kubectl get nodes -o json | jq '
  [.items[].status.allocatable] |
  {
    cpu: [(.[].cpu | gsub("m";"") | tonumber) / 1000] | add,
    memory_gi: [(.[].memory | gsub("Ki";"") | tonumber) / 1048576] | add
  }'

# Currently requested resources (sum of all pod requests)
kubectl get pods -A -o json | jq '
  [.items[].spec.containers[].resources.requests // {}] |
  {
    cpu_requested: [.[].cpu // "0m" | gsub("m";"") | tonumber] | add / 1000,
    mem_requested_gi: [.[].memory // "0Mi" | gsub("Mi";"") | tonumber] | add / 1024
  }'

# Headroom per node (allocatable - requested)
kubectl describe nodes | grep -A5 "Allocated resources:"

# Quick headroom summary script
kubectl get nodes -o custom-columns=\
"NAME:.metadata.name,\
CPU_ALLOC:.status.allocatable.cpu,\
MEM_ALLOC:.status.allocatable.memory,\
READY:.status.conditions[-1].type"