resource-tuning

Skill: Resource Tuning

Expertise: CPU/memory right-sizing, HPA, VPA, KEDA event-driven scaling, namespace quota design.

When to load

When pods are OOMKilled, CPU-throttled, underutilised, or autoscaling isn't working as expected.

Right-Sizing Methodology

1. Observe → 7-day peak metrics (kubectl top / Prometheus)
2. Set request = average × 1.1 (room for normal variance)
3. Set limit = p99 peak × 1.3 (room for spike without OOM)
4. Verify no throttling with: throttled_cpu_seconds metric
5. Adjust after 2 weeks of production data

# Current resource usage (snapshot)
kubectl top pods -n <ns> --sort-by=memory
kubectl top pods -n <ns> --sort-by=cpu

# Historical usage via Prometheus (7-day p99)
# CPU p99:
rate(container_cpu_usage_seconds_total{namespace="<ns>",pod=~"my-app-.*"}[5m])
  > quantile_over_time(0.99, rate(...[5m])[7d:5m])

# Memory p99:
quantile_over_time(0.99,
  container_memory_working_set_bytes{namespace="<ns>",pod=~"my-app-.*"}[7d:5m])

HPA Configuration

# CPU + Memory HPA (Kubernetes 1.23+)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70    # scale out at 70% avg CPU
    - type: Resource
      resource:
        name: memory
        target:
          type: AverageValue
          averageValue: 400Mi       # scale out if avg pod memory > 400Mi
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # wait 5 min before scaling down
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60             # scale down max 1 pod per minute
    scaleUp:
      stabilizationWindowSeconds: 0    # scale up immediately
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15            # can double every 15 seconds