Design GPU model inference serving

LLM inference has two phases with opposite profiles. Prefill processes all prompt tokens in parallel; it is compute-bound and fast per token, and it determines time to first token. Decode generates one token per step for each sequence; every step must read the full model weights from GPU memory, so at small batch sizes it is memory-bandwidth-bound and the GPU's compute sits idle. The fix is batching many sequences per step so one weight read serves many tokens. But each running sequence keeps its KV cache (the attention keys and values for every past token) in GPU memory, about 0.31 MB per token for a 70B model with grouped-query attention. A node with ~500 GB free for KV holds ~1.6M tokens, so batch size, and therefore throughput, is capped by memory, and long-context requests are expensive because they hold memory for their whole life. This is why quantising weights to FP8 (more room for KV), quantising the KV cache itself, and limiting max context per feature directly raise throughput. It also explains the SLOs: TTFT is governed by prefill and queueing; time per output token by batch size and model size. Pushing batch size up raises throughput but slows every user's token rate, so the scheduler caps the batch to keep TPOT under 50 ms.

Static batching (wait for N requests, run them together until all finish) wastes the GPU: short answers finish early and their slots sit idle while the longest request completes, and new requests wait for the next batch. Continuous (iteration-level) batching, popularised by Orca and vLLM, re-forms the batch at every decode step: finished sequences leave, waiting ones join immediately. Memory management makes this possible. Pre-allocating max-context KV for every request wastes most of it; PagedAttention stores KV in fixed-size blocks (e.g. 16 tokens) allocated on demand, like virtual memory pages, so fragmentation is small and identical prefixes can share blocks. Prefix caching builds on this: requests that start with the same system prompt and tool definitions reuse the cached blocks and skip that part of prefill, often saving 30-70% of prefill work for products with long fixed prompts. Chunked prefill splits a long prompt into pieces mixed with decode steps, so one 30k-token document does not freeze every other user's stream. When KV memory runs out, the engine preempts the lowest-priority sequences (drop and recompute later, or swap to CPU memory), which is how batch traffic yields to interactive traffic.

Requests differ wildly in cost: a 200-token chat and a 20k-token summary are both 'one request'. Round robin therefore overloads some replicas while others idle. The router should balance on real load signals pushed by replicas every second: waiting queue length, running sequences, and KV-cache usage. It should also be cache-aware. If requests for the same feature (same system prompt, same tools) go to the same few replicas, their prefix is already cached and prefill is much cheaper; scattering them destroys the cache. A practical approach is consistent hashing on a hash of the prompt prefix with bounded load: go to the hashed replica unless it is above, say, 1.25× average load, then try the next one on the ring. For LoRA fine-tunes, add adapter affinity: route tenant42-v3 to replicas that already hold that adapter, loading it elsewhere only when needed; engines such as vLLM and LoRAX can batch requests for different adapters on the same base model, so hundreds of fine-tunes share one GPU pool instead of each needing its own. Stickiness for multi-turn chat (same conversation → same replica) also reuses the conversation's KV cache between turns.

CPU utilisation tells you nothing here, and even GPU utilisation is misleading because a memory-bound decode can show high utilisation at low throughput. Scale on the signals users feel: queue wait time, TTFT p95, and KV-cache usage. Cold start is the hard part: a 70B replica needs nodes (may take minutes if capacity is scarce), ~70 GB of weights and warm-up, so 2-5 minutes in total. Reactive scaling alone will breach the SLO during a sudden spike. Combine three tools. Predictive scaling from daily and weekly patterns (Indian traffic has a clear evening peak, and sales and cricket matches are scheduled). A warm buffer of 10-15% spare capacity. And a fast safety valve: batch traffic on the same pool is paused within seconds, and overflow interactive traffic can go to an external provider. Speed up cold starts by caching weights on node-local NVMe, using a fast loading format (safetensors with parallel reads), and keeping CUDA graphs and images pre-built. Scale down slowly, during troughs, and fill night-time capacity with batch jobs (offline evals, embedding backfills, summaries) so reserved GPUs are not idle; reserved or committed GPUs for the base load are far cheaper than on-demand, and spot GPUs fit only the preemptible batch tier.

External APIs need no GPUs or MLOps team and offer the strongest models. Self-hosting wins on cost at high, steady volume, keeps data in India for regulated workloads, and allows fine-tunes and custom latency targets. Break-even depends on utilisation: idle reserved GPUs are pure loss, so most teams self-host the high-volume, well-defined workloads and keep an external provider for hard tasks and overflow.

TP=8 across one node gives the most KV memory per replica and lowest per-token latency, but every step needs all-reduce over NVLink, and one GPU failure takes the whole replica down. Two TP=4 replicas per node give more independent replicas and smaller failure domains with slightly higher latency. Never stretch TP across nodes for interactive traffic unless the interconnect is very fast.

Bigger batches give more tokens per GPU-second (lower cost) but slower tokens per user. Separate pools or priority classes let batch traffic run with large batches and interactive traffic with capped batch size and SLO-driven admission.

FP8 weights roughly halve memory and speed up both phases on H100-class GPUs with small quality loss; INT4 saves more but hurts quality on reasoning tasks. Decide per model with an evaluation run, not by default.

A few 100k-token requests can consume a replica's KV memory and block hundreds of short requests. Cap context per feature, route long-context traffic to a dedicated pool, and use chunked prefill so long prompts do not freeze decoding for others.

RAG features with 8k-token prompts make prefill the bottleneck and TTFT spikes. Raise prefix-cache hit rates through stable prompt ordering (static parts first), and at larger scale consider disaggregated serving: separate prefill and decode GPU pools with KV transfer between them, as in NVIDIA Dynamo and similar designs.

An ECC error or NCCL hang on one GPU stalls the whole TP=8 replica. Health checks run a tiny canary generation every few seconds with a timeout; on failure the router stops sending traffic, in-flight requests fail fast with a retryable error (the gateway retries once on another replica if no tokens were streamed yet), and the node is drained and replaced.

A new quantised build produces subtly worse answers. Canary at 1-5% traffic with automatic comparison of evaluation scores and error rates from the eval platform; rollback is a router weight change because the old replicas stay up until the canary is promoted.

Admission control protects SLOs for admitted requests: interactive requests beyond capacity go to the external provider or get a fast 429 with Retry-After; batch work waits. Never let queues grow unbounded, because every queued request already holds a client connection.