Design a queue-based autoscaling worker platform

CPU-based HPA is the default in Kubernetes and the wrong one for most queue workers. A worker waiting on an LLM or an HTTP API uses maybe 5% CPU even when 2M jobs are waiting, so CPU scaling never triggers; a CPU-heavy image worker may show 90% CPU while the queue is empty. Queue backlog is the honest signal of waiting work. The simplest form is backlog per replica: desired = ceil(lag ÷ target), where target is how many messages one replica can clear in an acceptable time. A better form reasons in time. With arrival rate λ jobs/s and processing time W s per job, Little's Law says you need λ × W jobs in flight just to keep up; to also clear a backlog B within deadline D you need extra (B ÷ D) × W. Divide by concurrency per pod to get pods. For example λ = 200/s, W = 0.7 s, B = 300k, D = 600 s gives 140 + 350 = 490 in flight, or 25 pods at 20 each. Also scale on the age of the oldest message for latency-sensitive queues (OTP notifications), because a small but stuck backlog matters more than a large fresh one. KEDA supports Kafka lag, SQS depth and arbitrary Prometheus queries, so the platform can compute this formula and expose it as one metric.

Adding workers only helps if the workers are the bottleneck. Three common ceilings make extra pods useless or harmful. First, Kafka partitions: consumers in one group cannot exceed the partition count, so the 49th pod sits idle; set maxReplicas to the partition count, create topics with enough partitions for peak (repartitioning later reshuffles keys), or use per-message concurrency inside each pod (e.g. a parallel consumer that processes keys concurrently while keeping per-key order). SQS has no such limit. Second, downstream rate limits: an LLM provider limit of 20M tokens per minute caps 800-token jobs at about 25,000 per minute regardless of pods; scaling to 200 pods just produces 429s and retries that waste money. Use a shared token bucket across pods and cap replicas at limit ÷ per-pod throughput. Third, shared databases: 100 pods × 10 connections = 1,000 connections can exhaust Postgres; use PgBouncer and size pools from the database limit backwards. A strong design lets each worker type declare its downstream limit, and the platform clamps maxReplicas accordingly, surfacing 'throughput-limited by provider' in the dashboard.

A naive autoscaler oscillates: lag spikes, it adds 40 pods, lag drops, it removes them, the next burst arrives. Each change in a Kafka consumer group triggers a rebalance; with the old eager protocol every consumer stops, gives up all partitions and waits, so frequent scaling makes throughput worse. Use cooperative-sticky assignment (or Kafka's newer consumer rebalance protocol) so only moved partitions pause, and static membership so restarted pods reclaim their partitions without a rebalance. On the HPA, scale up fast (allow +100% every 15 s) and scale down slowly (stabilisation window 5 minutes, remove at most 10-20% per minute). Scale to zero saves money for idle queues but adds cold start: KEDA's poll interval, pod scheduling, image pull, client warmup, and possibly a new node, often 1-3 minutes in total. For queues with tight deadlines, keep minReplicas at 1-2 and keep a small buffer of warm node capacity (low-priority placeholder pods that get preempted). Pre-pull large images with a DaemonSet or use smaller images. For predictable peaks such as an 8 PM sale start or IPL match, schedule a pre-scale 10 minutes before (KEDA cron scaler) instead of reacting after the queue builds.

Lambda with SQS scales per message automatically, has no nodes to manage and scales to zero for free, but has a 15-minute limit, cold starts, per-invocation cost that gets expensive at sustained volume, and concurrency limits per account. KEDA on Kubernetes needs more setup but suits long jobs, sustained high volume, custom images and spot pricing. Many companies use both: serverless for spiky low-volume queues, KEDA for heavy ones.

Spot is ~60-80% cheaper but can be reclaimed with 2 minutes' notice. Queue workers are ideal for spot if jobs are short or checkpointed and idempotent. Keep latency-critical queues (OTP, payment notifications) on on-demand or with an on-demand minimum, and use many instance types to reduce interruption risk.

A shared cluster with priority classes and quotas packs work tightly and cuts cost, but needs fairness rules so one team's burst does not starve others. Priority classes let payment workers preempt bulk jobs; ResourceQuotas cap each team.

Launching 100 nodes takes minutes and spot capacity in one instance type can run out. Allow many instance families and sizes, multiple AZs, and keep a warm buffer; predictable events get scheduled pre-scaling.

If the scaler reads lag from Prometheus with a 60 s scrape interval plus a 15 s KEDA poll, reaction is slow. Use direct scalers (KEDA reading Kafka/SQS directly) for the primary signal and shorter intervals for latency-critical queues.

A message that always fails keeps lag high, so the autoscaler adds pods that cannot help. Cap retries, move failures to retry topics with delays and then a DLQ, and alert on DLQ growth.

The LLM API returns 503 for 10 minutes. Workers trip a circuit breaker and stop consuming (or pause the worker via the API) instead of retrying in a tight loop. Lag grows, but the autoscaler must not add pods while the breaker is open; scaling is suppressed by a 'downstream unhealthy' metric. When it recovers, the backlog drains within the provider's limit.

Offsets for in-flight jobs were not committed, so they are redelivered. Idempotency keys make re-processing safe. Commit offsets only after the side effect succeeds.

Replicas stay at their current count; nothing breaks immediately, but bursts are not absorbed. Alert on oldest-message age as a backstop, and keep sensible minReplicas for critical queues.