Streaming LLM responses at scale

An LLM writes its answer one token at a time. Without streaming, the user stares at a spinner for the full 5 to 30 seconds. With streaming, text starts appearing after the time to first token (TTFT), often under a second, and keeps flowing. Users judge AI features by TTFT much more than by total time.

Server-Sent Events (SSE) is the usual transport: a normal HTTP response with Content-Type text/event-stream that stays open while the server writes small 'data:' events. It is one-way (server to client), works through most proxies and HTTP/2, and has a standard reconnect mechanism with Last-Event-ID. Model providers stream to your backend the same way. WebSockets are two-way and are worth their extra operational cost only when the client also sends frequent messages, such as voice or collaborative editing.

At scale the challenge is not the protocol but the shape of the load. A request that used to last 50 ms now lasts 20 seconds, so the number of open connections, the memory per connection, proxy timeouts, slow mobile clients and what happens when a connection drops halfway become the main design problems.

1. 1Client sends the message with POST (the browser's EventSource API only supports GET without a body or custom headers, so chat apps usually use fetch() and read the response body as a stream, parsing the SSE format themselves).
2. 2The server persists the user message, assigns a message id for the reply, and opens a streaming call to the AI gateway, which opens one to the provider.
3. 3The server responds with 200, Content-Type text/event-stream, Cache-Control no-cache, and disables proxy buffering (for Nginx, the X-Accel-Buffering: no header or proxy_buffering off). Buffering proxies hold the whole response and destroy streaming.
4. 4For each chunk from the model, write an event with an increasing id and flush immediately. Batch tiny tokens into a write every 30 to 50 ms if needed; it cuts syscalls and network packets with no visible delay.
5. 5Send a heartbeat comment (a line starting with ':') every 10 to 15 seconds while waiting, because load balancers and proxies close idle connections (AWS ALB and Nginx both default to 60 seconds).
6. 6Bound everything: each connection gets a small bounded buffer. If a slow client cannot keep up, either drop it or switch it to catch-up mode; never let one slow reader block a shared loop or grow memory without limit.
7. 7Propagate cancellation: when the client disconnects or presses Stop, cancel the request context so the gateway closes the provider stream. Every token generated after that is money spent for nothing.
8. 8Finish cleanly: send a final event with the finish reason and token usage, persist the full assistant message, and record TTFT, tokens per second and total time.
9. 9Plan for drops: either the generation is tied to the connection (cancel on disconnect, simplest and cheapest), or generation runs detached and appends chunks to a store such as a Redis Stream keyed by message id, so a reconnecting client asks for 'everything after event id 412' and continues.

func streamChat(w http.ResponseWriter, r *http.Request) {
    flusher, ok := w.(http.Flusher)
    if !ok {
        http.Error(w, "streaming unsupported", http.StatusInternalServerError)
        return
    }
    ctx := r.Context() // cancelled when the client disconnects
    tokens, err := gateway.Stream(ctx, buildRequest(r))
    if err != nil {
        http.Error(w, "model unavailable", http.StatusServiceUnavailable)
        return
    }
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("X-Accel-Buffering", "no") // stop Nginx buffering

    ping := time.NewTicker(15 * time.Second)
    defer ping.Stop()
    for seq := 1; ; {
        select {
        case <-ctx.Done():
            return // gateway.Stream sees ctx cancel and closes the provider call
        case <-ping.C:
            fmt.Fprint(w, ": ping\n\n")
            flusher.Flush()
        case t, open := <-tokens:
            if !open {
                fmt.Fprint(w, "event: done\ndata: {}\n\n")
                flusher.Flush()
                return
            }
            fmt.Fprintf(w, "id: %d\ndata: %s\n\n", seq, t.JSON())
            flusher.Flush()
            seq++
        }
    }
}