Design a high-concurrency streaming API server

Two hundred thousand mostly idle connections are cheap only if per-connection state is small. One OS thread per connection (classic Tomcat with blocking IO) costs ~1 MB of stack each and context switching, so it tops out at a few thousand. Use an event-loop or lightweight-task model: Go goroutines (a few KB of stack, multiplexed on OS threads with epoll), Rust Tokio tasks, Node.js, or Java virtual threads / Netty. Then bound everything, because unbounded anything becomes the outage. Per instance: an admission semaphore capping active streams (e.g. 20,000), and a fast rejection when it is full. Per connection: a small send buffer or channel (e.g. 64 events); the writer never blocks the producer. Per downstream: fixed connection pools (DB, Redis) and an HTTP/2 client to the LLM gateway with a maximum number of concurrent streams. Every operation has a timeout: read-header timeout (stops slowloris), idle timeout, and per-call deadlines carried in context. Raise OS limits: file descriptors (ulimit -n), somaxconn, ephemeral port range for upstream connections, and tune TCP buffer sizes, since default kernel buffers per socket can add up to gigabytes at 200k connections. Measure with a load test at rising concurrency and watch memory, goroutine count and open FDs, not just latency.

A client on a weak 2G or crowded stadium network may read slower than tokens are produced. If the server buffers without limit, memory grows until the process is killed and every user on that instance loses their stream. With per-connection bounded buffers, a full buffer means the client is too slow: for token streams, coalesce (merge queued deltas into one larger event, which is lossless because order is kept), and if the client is still behind after a timeout (e.g. 30 s with no progress), close the connection and let it resume from the Redis buffer later. For status updates, drop intermediate states and send the latest. Overload at the instance level is handled by admission control, not queues: when active streams reach the cap, new requests get 503 with Retry-After immediately, so admitted users keep a flat inter-token latency instead of everyone getting slow (the latency collapse of an overloaded server). Load balancers should spread by least connections, not round robin, because streams are long-lived and uneven. Also shed load upstream: if the LLM gateway returns 429 (provider rate limit), reject new generations early rather than letting them queue for minutes. Expose shed counts by reason so capacity planning is based on data.

The simplest design proxies the LLM stream directly to the client connection. It has two problems. A network blip kills the generation, so the user must start over and you pay for the tokens twice; and a disconnect is the only way to cancel. Decoupling fixes this: a generation worker owns the upstream call and appends tokens with sequence numbers to a short-lived buffer (Redis Streams, NATS JetStream, or in-memory with sticky routing for simpler setups). Any stream server can serve any reader from any sequence number, so SSE's built-in Last-Event-ID reconnect gives real resume, and the load balancer needs no sticky sessions. Cancellation becomes explicit: the user presses stop (POST :cancel), or no reader has been attached for N seconds, and the worker cancels the upstream request's context, which closes the HTTP/2 stream to the provider and stops token billing. Persist the final message once at completion; persisting every token to Postgres would turn 6 million events per second into database writes. The cost is an extra hop (~1 ms within a zone) and a Redis tier sized for the write rate; batching tokens every 30-50 ms keeps it manageable without users noticing.

Streams that last 60 seconds make naive rolling deploys drop users. Graceful shutdown in order: on SIGTERM, flip /readyz to failing so the load balancer stops sending new connections (wait ~5-10 s for it to notice), stop accepting new streams, let active streams finish up to a deadline (e.g. 90 s, matching terminationGracePeriodSeconds), then send remaining clients a 'reconnect' event or just close, and they resume on another instance from the Redis buffer with Last-Event-ID. Because generation lives in workers decoupled from stream servers, restarting a stream server never loses work. Generation workers drain the same way: finish current generations, take no new ones. Roll out slowly (maxUnavailable 10%) so the remaining instances can absorb reconnects, and watch for reconnect storms: thousands of clients reconnecting at the same moment should use jittered backoff, and SSE's retry field lets the server set it. For WebSocket-based designs the same applies, plus a protocol-level 'go away' message.

SSE is plain HTTP, works through most proxies, has built-in reconnect with Last-Event-ID, multiplexes over HTTP/2, and fits one-way token streams. WebSockets are bidirectional and needed for frequent client messages (collaboration, voice), but need custom reconnect/resume, sticky handling and more proxy care. Long polling works everywhere but adds latency and request overhead per token batch. SSE is the default for LLM streaming; use HTTP/2 to avoid the browser limit of ~6 connections per domain under HTTP/1.1.

Direct proxying is simpler and has one less hop and component, fine for small scale. The buffer adds Redis cost and complexity but gives resume, explicit cancel, stateless stream servers and safe deploys. At 200k concurrent mobile users, resumability pays for itself.

One event per token gives the smoothest typing effect but multiplies syscalls, Redis writes and bandwidth. Batching every 30-50 ms is visually the same to users and cuts writes by 2-5x.

If an AZ fails, ~70k clients reconnect at once and TLS handshakes are CPU-heavy. Terminate TLS at the load balancer or use session resumption, keep spare capacity (N+1 AZ), and use jittered client retry.

Millions of tiny XADDs per second can saturate Redis. Batch tokens, shard by generation_id across a cluster, and keep entries small; for very large scale, keep the buffer in the generation worker's memory with consistent routing and use Redis only as a fallback.

The real limit is often the provider's tokens per minute. Admission control should use the LLM gateway's budget signal so the server rejects early instead of holding thousands of connections waiting for capacity.

Its clients' connections drop; they reconnect (to other instances) with Last-Event-ID and resume from Redis. No generation is lost because generations run in workers.

The stream shows no progress; a watchdog sees the generation's heartbeat expire, marks it failed, and the client gets an error event with a 'regenerate' option. Billing uses only reported usage.

The gateway's circuit breaker fails over to a backup model; if none is available, new generations are rejected quickly with a clear error, and active streams get a timeout error event instead of hanging forever (per-call deadline).