Design an LLM gateway with failover and budgets

You cannot know the output tokens before the model answers, and a streaming call can run for 30 s. Checking 'spent < limit' and adding the cost afterwards lets 2,000 concurrent requests all pass the check at 99% and overshoot badly. Use reserve-then-settle. Before the call, count input tokens exactly with the model's tokenizer and reserve input_cost + max_tokens × output_price in Redis with a Lua script that atomically checks spent + reserved + this ≤ limit. After the call, settle: subtract the reservation, add the real cost from the provider's usage field (which also reports cached input tokens at their discounted price). Reservations carry a TTL-backed record so a crashed gateway does not leak reserved money forever; a sweeper releases reservations older than the max call duration. This bounds overshoot to roughly the in-flight error of a few requests. Keep two levels: a soft limit that alerts or downgrades the model, and a hard limit that returns 429. Also enforce provider-side limits (tokens per minute per deployment) separately, because those protect your rate quota, not your money. Use the provider's returned usage for billing, never only your own estimate.

Failover has three traps. First, mid-stream failure: once you have sent tokens to the caller you cannot transparently retry, because the second model will write a different answer. So retry or fail over only before the first token; after that, end the stream with an error event and let the caller decide. Use a separate TTFT timeout (e.g. 4 s) and a total timeout (e.g. 60 s) so a hung provider is detected early. Second, which errors to fail over on: 429 and 5xx and timeouts, yes; 400 (bad request, context too long) and content-policy refusals, no, because the fallback will fail the same way or behave differently. Third, model equivalence: a prompt tuned for model A may produce worse output on B. Only list fallbacks that passed the feature's offline eval set, and store the fallback chain per feature, not globally. Circuit breakers should be per deployment (provider + model + region + key), not per provider, so one bad Azure region does not take down the healthy ones. Add jittered backoff and a retry budget (retries ≤ 10% of traffic) so a provider brownout does not get a retry storm from 50 gateway pods.

Providers enforce requests-per-minute and tokens-per-minute per key, project or deployment. At 1,500 req/s with 2,300 tokens each you need about 200M tokens per minute, far more than one key allows. So the gateway holds a pool of deployments per model (several regions, several keys) and routes by remaining quota, not round robin. Each deployment has a token bucket in Redis refilled at its TPM; the router picks the deployment with the most remaining tokens whose breaker is closed, and treats provider 429s with a Retry-After as a signal to pause that deployment. Add priority: interactive features (chat a user is waiting on) get first claim on quota, while batch jobs (nightly summarisation) use a separate pool or the provider's batch API at a large discount. When total demand exceeds total quota, shed the batch traffic first. For India-residency tenants the candidate set is filtered to in-region deployments only, even if that means a 429 instead of failing over abroad; this is a compliance rule, not a performance choice.

Centralising every LLM call creates a new single point of failure, so the gateway must be boring and stateless. Run it as many pods across three zones behind an L4 load balancer, with config held in memory and refreshed by push, so Postgres being down does not stop traffic. Redis is the only hot-path dependency: give every Redis call a 5 ms timeout and a fallback. If Redis is unreachable, skip the cache and switch budgets to a local in-memory limit of (remaining budget ÷ number of pods) for a few minutes; that is fail-open with a bound, written down in an ADR. Size for streaming: 10,000 open connections means file descriptors, memory per stream and idle timeouts matter more than CPU. Use an async server (Go, Rust, or Envoy-based), cap in-flight requests per pod with a semaphore and return fast 503s when full. Propagate client disconnects to the provider call so an abandoned stream stops costing money. Deploy with canaries and drain connections gracefully on shutdown, because a rolling deploy otherwise cuts thousands of streams.

Open-source and managed gateways already give an OpenAI-compatible proxy, multi-provider routing and basic budgets. Start with one if your scale and compliance allow it. Build (or heavily extend) when you need tight integration with internal auth, residency rules, custom cost allocation, or very high concurrency. In the interview, say you would start from an existing proxy and explain which parts you would own.

A library in each service avoids a network hop (~1-2 ms, small next to a 2 s LLM call) but spreads keys, budgets and routing logic across 40 deployments and makes provider changes slow. A central gateway gives one place for keys, cost and failover. The hop cost is negligible for LLM traffic, so the gateway wins; keep a thin client SDK only for retries and tracing headers.

Exact caching is always correct but hits rarely (5-10%) because prompts include user-specific context. Semantic caching hits more but can return a wrong answer for a similar-looking question. Make semantic caching an opt-in per feature with a measured false-hit rate, scoped by tenant; see the semantic cache design for details.

Failing closed turns a Redis blip into an AI outage across the company. Failing fully open risks a runaway agent loop burning a day's budget. A local per-pod budget slice for a few minutes is the middle ground; for features with strict customer-paid quotas you may fail closed.

At 10× traffic the gateway scales horizontally, but provider TPM limits do not. Request quota increases early, add more deployments and regions, move batch work to batch APIs, route simple features to small or self-hosted models, and use prompt caching so repeated prefixes are cheaper and faster.

One team with a single feature doing 800 req/s hammers one Redis key with reserve and settle calls. Batch locally: each pod takes a lease of budget (e.g. ₹500) from Redis and spends it in memory, renewing when low. Overshoot is bounded by pods × lease size.

At 50M events/day the aggregator falls behind and dashboards go stale. Partition Kafka by team, batch inserts into ClickHouse (10k rows per insert), and pre-aggregate per minute with materialised views.

Breakers open within seconds, routes fail over to evaluated fallbacks, features with no fallback return a clear 503 and the product shows its non-AI fallback (e.g. 'summary unavailable'). Quota on the fallback provider becomes the constraint, so batch traffic is paused automatically.

Callers see a broken stream and retry with the same Idempotency-Key; reservations from the dead pod expire through the sweeper. Usage for the lost call may be missing, so a daily reconciliation compares gateway totals against provider usage exports.

A typo routes a high-traffic feature to an expensive model. Config changes are validated (model exists, price within allowed tier), rolled out to 5% of pods first, and a cost-per-minute anomaly alert triggers auto-rollback to the previous version.