Autoscaling and capacity planning

Capacity planning answers: how many servers, database connections, partitions and how much storage do we need for the next peak? You measure what one instance can handle at an acceptable latency (by load testing), forecast peak demand, and add headroom for failures and surprises.

Autoscaling changes capacity automatically based on a signal: CPU, requests per second per instance, queue depth or consumer lag, or a schedule. In Kubernetes the Horizontal Pod Autoscaler (HPA) scales pods, KEDA feeds it event signals like Kafka lag and can scale to zero, and the Cluster Autoscaler or Karpenter adds nodes when pods cannot be scheduled. On AWS, Auto Scaling groups do the same for EC2 instances.

The key limit is reaction time. The metric must rise, be collected (15 to 60 seconds), the autoscaler must decide, a node may need to boot (1 to 3 minutes), and the app must start and warm caches. Scaling from the alarm to useful capacity often takes 2 to 5 minutes. A traffic spike that triples load in 30 seconds, like a flash sale opening or a wicket in a cricket final, outruns reactive autoscaling. Those events are handled by planning and pre-scaling, with load shedding as the safety net.

Stateless tiers scale easily. Databases, Kafka partitions, connection pools and third-party quotas do not. Good capacity planning finds the component that will hit its limit first and plans for it, rather than only adding app servers.

1. 1Measure per-instance capacity with a load test that mirrors real traffic: find the request rate at which p99 latency crosses your target (say 300 ms) or errors appear. That is the instance's limit; plan to run at 50 to 70% of it.
2. 2Forecast the peak: from last year's peak and growth, marketing plans and event calendars. Use Little's Law (in-flight requests = arrival rate x latency) to size concurrency, threads and connection pools.
3. 3Compute instances needed = peak RPS / (per-instance RPS x target utilisation), then add N+1 or zone headroom so losing one availability zone does not overload the rest.
4. 4Check every downstream limit at that peak: database CPU and connections (instances x pool size must stay below max_connections or go through a pooler like PgBouncer), cache throughput, Kafka partitions (max useful consumers per group = partitions), and third-party rate limits (payment gateway, SMS provider, LLM tokens per minute).
5. 5Choose the scaling signal. CPU works for CPU-bound services. For I/O-bound services (waiting on databases or LLMs) use requests per second per instance or in-flight requests. For workers, use queue depth or consumer lag (via KEDA).
6. 6Configure the autoscaler: min and max replicas, target value, and stabilisation windows. Kubernetes HPA computes desired = ceil(current x currentMetric / targetMetric); by default it scales up quickly and waits 5 minutes before scaling down to avoid flapping.
7. 7Pre-scale for known events with scheduled scaling: raise min replicas an hour before, warm caches, increase database capacity and request higher provider quotas days before.
8. 8Keep the safety net: rate limits, load shedding and graceful degradation for when demand beats even the plan. Review after each event: forecast vs actual, first component to saturate, cost.