Real-time at scale: WebSockets, SSE and presence
In one line
Real-time systems keep millions of long-lived connections on a stateless-looking but connection-holding gateway tier, route messages to the right gateway through pub/sub and a session registry, and treat presence as cheap, approximate, TTL-based state.
What it is
Real-time features push data to users as it happens: chat messages, typing indicators, live order tracking, cricket scores, stock ticks, streamed LLM tokens. Plain request-response HTTP cannot push, so systems use one of three transports. Long polling: the client sends a request that the server holds open until data arrives. Server-Sent Events (SSE): one long HTTP response that streams events from server to client, with automatic reconnect and Last-Event-ID resume built into browsers. WebSockets: a full-duplex connection upgraded from HTTP, for frequent messages in both directions.
The hard part is not the protocol but the scale. Each connected user holds an open TCP connection on some server for hours. A product with 10 million concurrent users needs a fleet of connection gateways, a way to find which gateway holds a given user, a pub/sub layer to deliver messages to the right gateways, and a plan for deploys and failures that drop millions of connections at once.
Presence ('online', 'last seen', 'typing') is a special case: it changes constantly, every change could be fanned out to many watchers, and it does not need to be perfectly accurate. Good designs make presence approximate, TTL-based and lazily delivered.
Why interviewers ask
Chat, live comments, ride tracking, collaborative editing, multiplayer games and AI streaming all depend on it, so this topic appears inside many design rounds. Interviewers look for the separation of connection gateways from business logic, how a message finds the right connection, fan-out costs for large groups or live events, handling slow clients and reconnection storms, and when SSE is simpler than WebSockets. Knowing mobile realities (background apps, push notifications, flaky Tier-2/3 networks) separates practical engineers from diagram drawers.
How it works
- 1Choose the transport. SSE for server-to-client streams over plain HTTP (live scores, notifications, LLM token streaming); it works through most proxies and with HTTP/2 avoids the browser's 6-connections-per-domain limit of HTTP/1.1. WebSockets when the client also sends frequently (chat, games, collaborative editing). Long polling as a fallback for hostile networks.
- 2Build a dedicated connection gateway tier. Gateways only terminate connections, authenticate them, track subscriptions and forward messages; business logic lives in ordinary stateless services. A tuned gateway (epoll-based servers in Go, Java Netty, Erlang/Elixir, Rust) can hold hundreds of thousands of idle connections; memory per connection (buffers, TLS state) is usually the limit.
- 3Load balance at L4 or with an L7 balancer that supports upgrades, with long idle timeouts. Connections are sticky by nature: once opened, a connection stays on one gateway.
- 4Keep a session registry: user_id -> gateway_id (and device), in Redis or a similar store with a TTL refreshed by heartbeats. Services that need to reach a user look it up and send to that gateway.
- 5Or use topic-based pub/sub: each gateway subscribes to channels for the rooms or users it currently serves (Redis Pub/Sub, NATS, or a custom channel server); producers publish once per channel and every interested gateway receives it. Use a durable log (Kafka) or the database for messages that must not be lost, and pub/sub only as the fast delivery path.
- 6Make delivery reliable at the application level: each message has an ID and sequence number per conversation; clients acknowledge and, after reconnecting, ask for 'everything after sequence N' from storage. Pub/sub can drop messages; the sync-on-reconnect path fixes gaps.
- 7Keep connections alive with heartbeats (ping every 20-30 s) because mobile networks and NAT boxes silently drop idle connections. On mobile, when the app is in the background, the OS closes sockets; use push notifications (FCM, APNs) instead.
- 8Protect against slow clients: give each connection a small bounded send buffer; if it fills, drop non-essential updates (typing, presence), coalesce (send only the latest score), or disconnect the client so it resyncs.
- 9Plan for reconnection storms: deploys drain gateways gradually, sending clients a 'reconnect' hint; clients reconnect with exponential backoff and jitter; authentication on reconnect must be cheap (verify a token locally).
- 10Presence: clients heartbeat every ~30 s; the presence service stores user -> last_seen with a TTL of about 60-90 s. Offline is inferred from TTL expiry, not from a perfect disconnect event. Deliver presence only to users currently looking at a contact or channel, and batch changes, instead of broadcasting every change to everyone.
- 11For huge broadcasts (a live match with millions watching), do not fan out per user from one service. Publish once per topic, let each gateway receive one copy and write it to all its local subscribers, and sample or aggregate low-value events such as reactions.
Worked example · Live score and reactions for a cricket final
- Peak: 20 million concurrent app users on the match page. Score updates: about 1 per ball, plus commentary, roughly 1 message every 5 s. Reactions: users send emoji bursts, up to 2 million reactions per second at a six or a wicket.
- Transport: SSE for the score feed (server to client only, simple through CDNs and proxies); reactions are sent as small HTTP POSTs, batched by the client every 2 s.
- Gateways: at 200,000 connections each, 20M / 200k = 100 gateway servers, plus 50% headroom for failures and reconnect spikes = 150. At about 20 KB per connection, 200k connections use about 4 GB of memory per gateway.
- Score fan-out: the scoring service publishes one event to topic 'match:final'. Each of 150 gateways receives one copy (150 messages, not 20 million) and writes it to its local connections. Each gateway writes 200k x 1 message per 5 s = 40,000 writes per second, which is manageable.
- Reactions: 2 million per second are not broadcast individually. An aggregator (Flink or in-memory counters, sharded by match) sums them per second and publishes one 'reaction counts' event per second: the client animates emojis from counts. Fan-out stays 150 messages per second.
- Reconnect storm: a bad gateway deploy drops 4 million connections. Clients reconnect with 1-30 s jittered backoff, so about 400,000 reconnects per second for 10 s instead of 4 million at once; token verification is local, so the identity service is not hit.
- Tier-2/3 networks: clients that reconnect send Last-Event-ID; the gateway replays the last few score events from a small in-memory buffer, so users on flaky 4G do not miss a wicket.
Picture it
A large hotel switchboard. Each guest room has a phone line permanently connected to one of many switchboard operators (gateways). The front desk keeps a register of which operator handles which room (session registry). When a message for a room arrives, the desk sends it to the right operator, who passes it on. For an announcement to the whole hotel, the desk does not call every room itself; it tells each operator once, and each operator informs their own rooms. Whether a guest is 'in' is judged by when they last picked up the phone, not by a perfect record.
Key patterns
Connection gateway + pub/sub
When: Any real-time product beyond one server
How: Gateways hold connections and subscriptions; services publish to topics; each gateway gets one copy per topic and delivers locally.
Durable store + fast path
When: Messages must not be lost (chat, orders)
How: Persist first (database or Kafka) with a sequence number, push via pub/sub as a best-effort fast path, and resync from the store on reconnect.
Coalescing and aggregation
When: High-frequency updates where only the latest value or the total matters
How: Send the latest location or score instead of every change; aggregate reactions into counts per second.
Lazy presence
When: Online status for large contact lists or big groups
How: Heartbeat with TTL; subscribe to presence only for contacts on screen; batch updates; show 'last seen a few minutes ago' rather than exact times in large groups.
Real-world use
WhatsApp on Erlang
WhatsApp engineers reported in 2012 that they handled about 2 million concurrent TCP connections on a single server running Erlang on FreeBSD, an early demonstration of how far a tuned connection tier can go.
Slack's gateway and channel servers
Slack's engineering blog describes gateway servers that hold WebSocket connections and subscribe to channels, and channel servers, assigned by consistent hashing, that hold channel state and fan out messages to the gateways with subscribers.
Discord on Elixir
Discord runs its real-time gateway and per-guild processes on Elixir (Erlang VM) and has written about scaling large servers, where one message or presence change can reach hundreds of thousands of members, by batching and limiting presence updates.
Live cricket on JioHotstar
Cricket streaming in India reaches tens of millions of concurrent viewers. Alongside the video CDN, live scores, polls and reactions must be fanned out cheaply, which is why topic-level fan-out and aggregation of reactions matter.
LLM token streaming
Chat-style AI products stream model output token by token, commonly over SSE (OpenAI and Anthropic APIs both stream with SSE). The server must stream through without buffering and handle clients that disconnect mid-answer.
In a project
A food delivery app shows live order status and the courier's location on a map. Today the app polls GET /orders/{id} every 3 seconds; at a dinner peak of 1.5 million active orders this is 500,000 requests per second, mostly returning 'no change', and the database is struggling.
- Introduce an SSE endpoint GET /stream/orders/{id} served by a separate gateway tier; the app falls back to 15-second polling if SSE fails.
- Courier apps keep sending location every 5 s over HTTP to a location service, which writes the latest position to Redis and publishes to topic 'order:{id}' only for active orders.
- Order-status changes from the order service are published through the outbox and Kafka, then forwarded to the same per-order topic.
- Gateways subscribe to topics for the orders their connected clients watch, and unsubscribe when the last viewer leaves.
- Send each event with a sequence number; on reconnect the client sends Last-Event-ID and the gateway returns the current state from Redis rather than replaying history.
- Coalesce location updates per connection: if a client is slow, send only the latest position.
- When the app goes to the background, close the stream and rely on push notifications for key status changes (picked up, arriving, delivered).
Impact: Requests drop from 500,000 polls per second to about 300,000 updates per second sent only when something changes, and the database no longer serves status reads at all. Users see the courier move within about 1 second instead of up to 3.
Trade-offs
Use it when
- SSE for one-way streams: notifications, live scores, dashboards, LLM tokens
- WebSockets for frequent two-way traffic: chat, games, collaborative editing, trading terminals
- Long polling or slow polling as a fallback, or when updates are rare and connections are expensive
- Push notifications (FCM, APNs) for users whose app is in the background
Avoid it when
- Real-time push for data that changes every few minutes; polling with caching is simpler and cheaper
- WebSockets when only the server sends: SSE gives reconnect and resume for free over plain HTTP
- Putting business logic in gateways: deploys then drop connections, and gateways become hard to scale
- Relying on pub/sub alone for messages that must not be lost
- Broadcasting every presence or typing change to every member of large groups
Common mistakes
- Forgetting that connections are stateful: you cannot deploy gateways like stateless services without draining and jittered reconnects.
- No heartbeats, so NAT and mobile networks silently kill idle connections and users stop receiving updates.
- Unbounded per-connection buffers, so a few slow clients exhaust gateway memory.
- Fanning out to millions of users from one service instead of one copy per gateway per topic.
- Treating a disconnect event as reliable proof that a user is offline instead of using TTLs.
- No gap detection: messages lost during reconnect are never recovered because there are no sequence numbers.
- Ignoring backgrounded mobile apps and assuming sockets stay open.
Interview questions
Quick revision
- Long polling < SSE < WebSockets in capability; choose the simplest that fits.
- SSE: one-way, HTTP, auto-reconnect, Last-Event-ID; use HTTP/2.
- WebSockets: full duplex, for chat, games, collaboration.
- Separate gateway tier holds connections; services stay stateless.
- Session registry (user -> gateway) or topic pub/sub routes messages.
- Persist first, push second; sequence numbers and resync on reconnect.
- Heartbeat every 20-30 s; mobile background = push notifications.
- Bounded buffers; coalesce or drop for slow clients.
- Jittered reconnects and draining for deploys.
- Presence = heartbeat + TTL, delivered lazily.
- Big broadcasts: one copy per gateway per topic; aggregate reactions.
Check yourself
1. Which transport is the best fit for streaming LLM output tokens to a browser?
2. A score update must reach 10 million viewers connected to 100 gateways. How many copies should the publisher send through pub/sub?
3. Why do real-time clients send heartbeats every 20-30 seconds?
4. What is the most robust way to decide that a user is offline?
See it in a full design
These walkthroughs use this lesson.
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.