Design a ChatGPT-like chat service

Generation has two phases. Prefill processes the whole prompt in parallel; it is compute-bound and sets time to first token. Decode produces one token at a time per request; it is memory-bandwidth-bound because every step reads the model weights and the request's KV cache (the attention keys and values of all previous tokens). Serving one request at a time wastes the GPU, so servers use continuous batching: new requests join the running batch at each decode step instead of waiting for the batch to finish (vLLM, TensorRT-LLM, SGLang do this). The limit is KV-cache memory: a long conversation can hold gigabytes, so paged KV cache (allocating it in blocks, like virtual memory) avoids fragmentation and lets more requests fit. Prefix caching reuses KV blocks for identical prefixes such as the shared system prompt and the earlier turns of the same conversation, which is why the scheduler should send a conversation back to the GPU that served its last turn. Large models are split across GPUs with tensor parallelism inside a node. Some fleets separate prefill and decode onto different GPU pools so long prompts do not stall streaming for everyone else. Capacity planning is in tokens per second per GPU at a target latency, not requests per second.

GPUs are the scarce, expensive resource, and you cannot autoscale them in seconds the way you scale web pods. The scheduler keeps a queue per (model, tier) and admits work to a GPU only when it has KV-cache room and the server's tokens-per-second per stream stays above the target (say 30 tok/s). Under pressure, apply policy in order: paid users first; free users wait in a visible queue ('high demand, your answer will start shortly'); then free users are routed to a smaller model; then free sign-ups or new conversations get rate-limited harder. Plan limits ('80 large-model messages per 3 hours') are themselves a capacity tool, enforced with sliding window counters before anything is queued. Set a maximum queue time (e.g. 20 s); requests older than that are rejected rather than served late, because a user who already left wastes GPU time. Cancel propagation matters: when a user presses stop or closes the app before the first token, remove the request from the queue or batch immediately. Track queue time, TTFT and tokens/s per tier as the real SLOs, and scale the fleet on forecasts (daily curve, exam season, launches) plus reserved capacity across regions.

If tokens flow directly from the GPU server to the user's TCP connection, every network blip kills the answer and costs a full regeneration, and every edge pod must be sticky to a GPU. Instead, the GPU server writes tokens to a per-generation Redis Stream (append-only, ordered, with entry ids), and any edge pod can read it with XREAD from a sequence number. SSE fits well: one-way server-to-client, works over HTTP/2 through proxies, and has Last-Event-ID built in for resume. The edge pods are tuned for long-lived connections: async I/O, heartbeats every 15 s so mobile carriers and proxies do not drop idle streams, proxy buffering disabled, and a bounded buffer per connection so a slow client cannot grow memory. At ~300k concurrent streams you need hundreds of edge pods and must drain them gracefully on deploy, which is easy here because clients simply reconnect and resume. Write coalescing helps cost: append every 3-5 tokens or every 50 ms rather than every token, reducing Redis operations by ~4× with no visible difference. The final message is persisted once at the end, not per token.

Models are stateless; every turn resends the conversation. The context builder walks the selected branch from the leaf back to the root, then fits it into a token budget: system prompt, user memory facts, attachment text, then turns from newest to oldest. When history is too long, keep the first message (it often states the task) and the recent turns, and replace the middle with a running summary generated asynchronously by a small model after each turn. Count tokens with the model's tokenizer; Indian-language text can use 3-5× more tokens than English for the same meaning, which matters for both limits and cost. Storage is simple key-value at huge scale: messages partitioned by conversation_id in Cassandra or DynamoDB, conversations partitioned by user_id sorted by updated_at for the sidebar. The tree model (parent_id per message) makes edit and regenerate cheap and keeps shared links stable. Writes per message are few (user message, assistant placeholder, final assistant message), so 500M messages/day is about 17k writes/s, comfortably handled by a modest Cassandra cluster. Deleting a conversation must also delete attachments, memory derived from it and cached summaries, which privacy laws such as India's DPDP Act expect.

SSE is one-way, plain HTTP, proxy-friendly and resumable with Last-Event-ID, which matches token streaming. WebSocket is needed for two-way low-latency traffic such as voice mode. Use SSE for text chat and add WebSocket or WebRTC only for voice.

Calling a provider API removes the whole inference plane from your design and is right for most companies. Running your own fleet only pays off at very large, steady volume or when you own the model, and it brings GPU procurement, scheduling and utilisation problems. The interview often asks for the own-fleet version to test that knowledge.

Postgres is fine up to a few thousand writes per second with sharding by user. At ~17k writes/s and petabyte-scale history, a wide-column store partitioned by conversation gives linear scaling and simple access patterns; you lose ad-hoc queries, which go to the analytics pipeline instead.

Traffic can double in an hour. Keep 20-30% headroom, have pre-approved degradation steps (shorter max output for free tier, smaller model, waiting room), and borrow capacity from batch or evaluation jobs that can be pre-empted.

Power users with 200-turn chats send huge prompts every turn. Summarise aggressively, rely on prefix caching with conversation affinity, and cap context per plan.

The scheduler sees the lost heartbeat, marks the generation failed at its last sequence, and either retries on another server from the start (sending a 'restart' event so the client clears partial text) or ends with an error and a regenerate button. Partial text already persisted in the Redis stream is not lost.

In-flight generations on that shard lose resume ability; clients still connected keep receiving after replica promotion, others see an error and can regenerate. Final messages are persisted to Cassandra, so history is safe.

Sending still works if the user message write is acknowledged by a quorum; reads fall back to the client's local cache of recent conversations. Never block streaming on the history write path.