cloud-native-event-driven-scaling

Event-Driven Scaling with KEDA

Purpose

CPU-based HPA reacts late to queue backlogs and cannot scale from zero. KEDA feeds event-source metrics to the HPA and handles 0↔1 activation itself, so consumers scale with work waiting rather than with CPU after the fact.

Agent Protocol

Trigger

"keda", "scale to zero", "ScaledObject", "ScaledJob", "scale on queue length", "kafka consumer autoscaling", "sqs scaling", "event driven autoscaling", "TriggerAuthentication".

Input Context

• Event source and auth model (IRSA/EKS Pod Identity, Workload Identity, secrets).
• Per-pod throughput (messages/s) and acceptable backlog drain time.
• Message processing duration (seconds vs hours).

Output Artifact

ScaledObject/ScaledJob + TriggerAuthentication YAML. No file unless requested.

Response Format

Resource: {ScaledObject|ScaledJob} (why)
Trigger: {type} target={value} activation={value}
Replicas: min={n} max={n} (max bounded by {partitions|downstream limit})
Flapping control: {cooldownPeriod, HPA behavior}
Auth: {TriggerAuthentication / ClusterTriggerAuthentication}

Completion Criteria

• Target value derived from per-pod throughput, not guessed.
• maxReplicaCount bounded by partitions or downstream capacity.
• Long-running work uses ScaledJob or graceful termination.

How KEDA Works

• keda-operator: watches ScaledObjects, polls scalers every pollingInterval (default 30 s), creates/owns an HPA, and performs activation (scales 0→1 when the metric exceeds activationThreshold/activation* value, and 1→0 after cooldownPeriod, default 300 s, with no activity).
• keda-operator-metrics-apiserver: implements the Kubernetes External Metrics API (external.metrics.k8s.io); the HPA queries it for 1↔N scaling.
• KEDA does not replace the HPA — scaling between 1 and N is the regular HPA algorithm: desired = ceil(current × currentMetric / target); for AverageValue external metrics this is effectively ceil(totalMetric / target).
• Only one external metrics server can be registered per cluster; KEDA conflicts with other adapters (e.g. prometheus-adapter for external metrics).