Intermediate 13 min read AI-era system design

The AI gateway: routing, failover and budgets

In one line

An AI gateway is the single internal service that every model call goes through, so routing, provider failover, per-tenant token budgets, caching, redaction and cost tracking are built once instead of in every team's code.

What it is

An AI gateway (also called an LLM gateway or LLM proxy) is a reverse proxy specialised for model APIs. Product services send it one internal request format, usually an OpenAI-compatible chat shape, with headers that say which tenant and which feature the call belongs to. The gateway decides which provider and model to use, calls it, streams the answer back and records what happened.

It is the AI version of the API gateway: one choke point for cross-cutting concerns. Typical responsibilities are authentication of internal callers, provider key management, routing by feature config, retries, circuit breaking and failover, per-tenant rate limits and token budgets, exact and semantic caching, PII redaction, guardrails, usage and cost events, and tracing.

You can build one (it is a few thousand lines in Go or Python) or run an existing one: open-source proxies such as LiteLLM, Envoy AI Gateway or Kong's AI plugins, or managed ones such as Cloudflare AI Gateway. The design questions are the same either way.

Why interviewers ask

Interviewers use the gateway to test whether you can turn a messy dependency into a controlled one. When a design has several AI features, saying 'all model calls go through an internal gateway' and then explaining failover, budgets and streaming passthrough in detail is a strong senior signal. It also shows up directly as a design prompt ('design an LLM gateway with failover and budgets') and in follow-ups like 'one customer's agent loop just burned your monthly budget, what stops it?'

How it works

1Accept a normalised request: POST /v1/chat with model alias (for example 'support-reply', not a vendor model name), messages, max_tokens, plus headers X-Tenant-Id, X-Feature and a request id. Callers never hold provider keys.
2Authenticate the caller (service identity via mTLS or a signed token) and load the feature config: primary model, fallback models, max input and output tokens, timeout, cacheable or not, allowed tenants.
3Check limits before spending: a token-bucket rate limit per tenant (requests and tokens per minute) and a budget counter per tenant per day or month in Redis. Reserve the estimated tokens up front; settle with the real usage the provider returns.
4Redact or tokenise personal data (emails, phone numbers, Aadhaar or PAN numbers) before the prompt leaves your network, and restore placeholders in the response if needed.
5Look up the cache: exact match first (hash of model, prompt version, messages and parameters), then optionally a semantic cache scoped by tenant and model.
6Route: pick the provider endpoint from the config, skipping any whose circuit breaker is open. Routing can also be by cost (cheap model for short prompts), by latency (fastest healthy region) or by data residency (an India-region deployment for regulated tenants).
7Call with a timeout on time to first token and on total time. On 429, 5xx or timeout before any token is streamed, retry with backoff and jitter or fail over to the next model. Once tokens have been streamed to the client, you cannot silently switch models; you end the stream with an error event and let the client retry.
8Stream tokens straight through (SSE) without buffering the whole answer, while counting tokens and watching for a client disconnect, which must cancel the upstream call to stop paying for unseen tokens.
9After the call, emit a usage event asynchronously (tenant, feature, model, input, cached and output tokens, cost, latency, cache hit, fallback used) to Kafka for billing and dashboards, and an OpenTelemetry span for tracing. Never block the response on this write.

Worked example · A primary provider outage during peak

Setup: the gateway handles 400 requests per second for 6 features. 'support-reply' is configured as primary = Model A on provider 1, fallback = Model B on provider 2, timeout to first token = 4 s.
19:02: provider 1 starts returning 503 for 30% of calls. Each failed call is retried once on provider 1 after about 200 ms with jitter, then sent to provider 2. Users see about 0.5 s of extra latency on those calls, no errors.
19:03: the circuit breaker for provider 1 trips when more than 50% of calls in a 10-second window fail (minimum 20 calls). For the next 30 seconds every 'support-reply' call goes straight to provider 2 with no wasted attempt.
Capacity check: provider 2 now receives the extra 120 requests per second x 3,000 tokens = 360,000 tokens per second (21.6M tokens per minute). The gateway's per-provider limiter shows provider 2's quota is 30M TPM, so it fits. If it did not, low-priority features (bulk summaries) would be shed first, keeping interactive features alive.
19:03:30: the breaker goes half-open and lets 5 probe requests through to provider 1. They fail, so it stays open for another 60 seconds (backoff on the breaker itself).
19:40: probes succeed, the breaker closes and traffic returns to provider 1 gradually (10%, then 50%, then 100%) so a half-recovered provider is not flooded.
Afterwards the usage events show 38 minutes of fallback traffic, its extra cost, and the eval score of Model B on this feature, which the team checked in advance.

Picture it

The gateway is like a company's travel desk. Employees do not book flights with their own cards; they tell the desk where and when. The desk knows which airlines have contracts, rebooks you on another airline when a flight is cancelled, enforces each department's travel budget and produces one monthly report of who spent what.

Reference code

async def route(req, cfg, breakers, budget):
    # cfg.models: ordered list, e.g. ["provider1/model-a", "provider2/model-b"]
    est = count_tokens(req.messages) + cfg.max_tokens
    if not await budget.reserve(req.tenant_id, est):          # atomic Lua in Redis
        raise TooManyRequests(retry_after=budget.reset_seconds(req.tenant_id))

    last_error = None
    for model in cfg.models:
        if breakers[model].is_open():
            continue
        try:
            stream = await call_provider(model, req, first_token_timeout=cfg.ttft_timeout)
            first = await stream.first_chunk()     # failover is only safe up to here
            breakers[model].record_success()
            return model, stream.prepend(first)
        except (RateLimited, ProviderError, Timeout) as e:
            breakers[model].record_failure()
            last_error = e
    await budget.release(req.tenant_id, est)
    raise AllModelsUnavailable(last_error)        # caller shows the degraded mode

Key patterns

Model aliases instead of model names

When: Many features, frequent model upgrades

How: Callers ask for 'support-reply'; the gateway config maps it to a concrete model and version. Upgrading a model is a config change plus an eval run, not a code change in every service.

Reserve then settle budgets

When: Token budgets with concurrent requests

How: Before the call, atomically reserve the estimated tokens (input count + max_tokens); after the call, refund the difference using the provider's reported usage. Prevents many parallel calls from overshooting the budget.

Failover only before the first token

When: Streaming responses

How: Retries and fallbacks are invisible only until bytes reach the client. After that, send an error event and let the client decide to retry, possibly with the partial text discarded.

Priority-based shedding

When: Provider quota or capacity shrinks

How: Tag each feature as interactive or batch. When quota is short, reject or delay batch calls first so user-facing features keep working.

Real-world use

Uber GenAI Gateway

Uber has described a central gateway that gives internal teams one interface to several external and self-hosted models, with shared authentication, PII redaction, logging and cost attribution.

LiteLLM proxy

A widely used open-source proxy that exposes an OpenAI-compatible API in front of 100+ providers, with fallbacks, per-key budgets, rate limits and spend tracking. Many teams start with it before building their own.

Cloudflare AI Gateway

A managed gateway that sits in front of model providers and adds caching, rate limiting, retries, model fallback and analytics without changing much application code.

Envoy AI Gateway

An open-source project built on Envoy Proxy for routing LLM traffic, with token-based rate limiting and provider fallback, aimed at Kubernetes platform teams.

In a project

A B2B SaaS company in Pune has five product teams each calling LLM providers directly with their own keys. Last month one enterprise customer's automation looped and spent about Rs 9 lakh in two days, and a provider outage took down three features at once.

Build a gateway service in Go with an OpenAI-compatible endpoint and SSE passthrough; move provider keys into it and revoke the teams' keys.
Define a config per feature in Git: primary and fallback models, max tokens, timeout, cacheable flag. Changes go through code review and roll out without deploys.
Add per-tenant token buckets (tokens per minute) and monthly budgets in Redis using a Lua script for atomic check-and-reserve; return 429 with Retry-After and a clear 'budget resets on' message.
Add a circuit breaker per provider and model, and a priority field so interactive features keep capacity when a provider is degraded and batch features are shed first.
Emit a usage event per call to Kafka; a consumer writes them to ClickHouse for per-tenant and per-feature cost dashboards and an alert when a tenant's hourly spend exceeds 3x its usual.
Run a fault-injection test monthly: make the primary provider return 503 for 10 minutes in staging and check that error rates stay under the SLO.

Impact: Runaway loops are stopped within minutes by the budget check, a later provider outage causes only a model switch visible on a dashboard, and finance gets exact cost per customer, which lets sales price the AI add-on correctly.

Trade-offs

Use it when

More than one team or feature calls LLMs
You need per-customer budgets, cost attribution or usage-based billing
You want provider failover or the freedom to switch models without code changes
Prompts may contain personal or regulated data that must be redacted or kept in-region

Avoid it when

A single prototype with one feature and no real users: a thin client library with a timeout is enough for now
Ultra-low-latency on-device inference where the call never leaves the phone
You would build a heavy synchronous pipeline (several network hops for cache, guardrail and logging) that adds hundreds of milliseconds; keep the hot path lean and do logging asynchronously

Common mistakes

Buffering the full response in the gateway before returning it, which destroys time to first token.
Retrying or failing over after tokens were already streamed, so the user sees two different half-answers stitched together.
Counting budgets with your own token estimate only, instead of settling with the usage figures the provider returns.
Assuming the fallback model behaves like the primary: it may need a different prompt and must pass the same evaluation set.
Not cancelling the upstream call when the client disconnects, so you pay for tokens nobody reads.
Making the gateway a single point of failure: it must be stateless, horizontally scaled and deployed in more than one zone, with Redis outages handled by a local fallback limiter.
Writing usage logs synchronously to a database on the request path.

Interview questions

Quick revision

One door for all model calls: auth, keys, routing, limits, cache, redaction, failover, usage events.
Callers use model aliases per feature; the config maps alias to concrete model and fallback list.
Budget: reserve input + max_tokens before the call, settle with provider-reported usage after.
Retry or fail over on 429/5xx/timeout only before the first streamed token.
Circuit breaker per provider+model: open on error rate, half-open probes, ramp traffic back gradually.
Check the fallback provider's TPM quota; shed batch features before interactive ones.
Stream through (SSE); cancel upstream when the client disconnects.
Usage and cost events go to Kafka asynchronously; never block the response.
Gateway is stateless and multi-zone; state lives in Redis with a local fallback limiter.
A fallback model needs its own prompt checks and must pass the eval set.

Check yourself

1. The gateway has already streamed 80 tokens to the client when the provider connection drops. What is the correct behaviour?

2. Why reserve input tokens plus max_tokens before calling the model?

3. Which is the best reason for callers to request a model alias like 'support-reply' instead of a vendor model name?

4. A provider's circuit breaker is half-open. What does that mean?

See it in a full design

These walkthroughs use this lesson.

Design an LLM gateway with failover and budgets Design a semantic cache for LLM calls Design a rate limiter service Design an AI customer support agent Design an LLM evaluation and observability platform Design a globally distributed rate limiter Design a high-concurrency streaming API server Design a real-time voice assistant Design GPU model inference serving Design a cost-per-request FinOps dashboard for AI features

Saved on this device. Create a free account to keep it on every device.

What changes when an LLM is in the system Exact, semantic and prompt caching

Practise this with Sam

A real mock interview, out loud, with an honest scorecard. Free to start.

Start free