What changes when an LLM is in the system
In one line
An LLM is a slow, expensive, rate-limited and non-deterministic dependency that sometimes fails or says wrong things, so you design around it the way you design around a flaky payment provider, not like a fast function call.
What it is
When a product calls a large language model (LLM), such as GPT, Claude, Gemini or a self-hosted Llama, it adds a dependency with very unusual numbers. A typical database query takes 1 to 10 ms. A typical LLM call takes 0.3 to 2 seconds before the first token arrives and 5 to 60 seconds to finish a long answer, because output is generated one token at a time (often 30 to 150 tokens per second).
The call is also priced per token, limited by the provider in requests per minute (RPM) and tokens per minute (TPM), and it can return HTTP 429 (rate limited) or 5xx errors during provider incidents. The same prompt can give different answers on different runs, and the answer can be confidently wrong (hallucination).
Finally, the model reads text as instructions. Any untrusted text you put into the prompt, such as a customer email, a web page or a support ticket, can try to change what the model does (prompt injection). That makes the prompt a security boundary, not just a string.
None of this needs new physics. It needs the classic tools applied deliberately: timeouts, queues, streaming, caches, budgets, circuit breakers, idempotency, least privilege and measurement. This module goes through each of them.
Why interviewers ask
Most product teams now ship at least one LLM feature, and interviewers increasingly add an AI twist to familiar prompts: 'design customer support, but with an AI agent' or 'add AI summaries to this feed'. They want to see that you size the system with LLM numbers (seconds and tokens, not milliseconds and rows), keep the user-facing path fast, control cost, handle provider failures and treat model output as untrusted. Candidates who draw the LLM as one more box with a 50 ms arrow are marked down quickly.
How it works
- 1Size with LLM numbers: for each feature estimate requests per second, input tokens (system prompt + context + user text) and output tokens per call. Tokens drive both cost and latency. A rough rule for English is about 4 characters or 0.75 words per token; many Indian languages take several times more tokens for the same meaning on common tokenizers.
- 2Apply Little's Law to concurrency: requests in flight = arrival rate x latency. 100 requests per second at 10 seconds each means 1,000 open requests, so connection limits, memory per request and database pool sizes matter more than CPU.
- 3Choose the interaction style: stream tokens to the user when a human is waiting (time to first token is what they feel), and move work to a queue or workflow when nobody is waiting or the job takes more than about 30 seconds.
- 4Put every model call behind one internal gateway: it holds provider keys, routes to models, retries and fails over, enforces per-tenant token budgets, caches, redacts personal data and records usage. Product services never call providers directly.
- 5Treat output as untrusted input: ask for structured output (JSON with a schema), validate it, and have a fallback when it does not parse. Never execute model output directly as SQL, shell or an API call without checks.
- 6Limit what the model can do: tools and agents get least-privilege credentials, allow-listed actions and human confirmation for risky steps, because prompt injection cannot be fully prevented.
- 7Design for failure: timeouts per call, a fallback model, a degraded mode (show the non-AI page, a cached answer or 'try again'), and idempotency keys so retries never double-charge or double-send.
- 8Measure quality, not just uptime: trace every call with the exact prompt, retrieved context, model version, tokens and cost, and run an evaluation set before every prompt or model change.
Worked example · Sizing an 'AI reply suggestion' feature for a support desk
- Requirement: when an agent opens a ticket, suggest a reply. 50,000 support agents, each opens about 80 tickets a day: 4M suggestions a day.
- Average rate: 4M / 86,400 = about 46 requests per second. Office-hours peak is roughly 3x: about 140 requests per second.
- Tokens per call: system prompt 800 + ticket history 1,500 + retrieved help articles 1,200 = 3,500 input tokens; the reply is about 300 output tokens.
- Latency: about 0.6 s to the first token and 300 tokens at about 60 tokens/s = 5 s to finish. Stream it, so the agent sees text within a second.
- Concurrency at peak (Little's Law): 140/s x 5.6 s = about 790 requests in flight. A server pool sized for 50 ms requests (7 in flight) would collapse; we need streaming-friendly servers and bounded admission.
- Provider throughput: 140/s x 3,800 tokens = 532,000 tokens per second = about 32M tokens per minute. That is far above the default limits of most accounts, so we need raised quotas, more than one provider or region, and caching.
- Cost at an illustrative price of $1 per million input tokens and $4 per million output tokens: 4M x 3,500 = 14B input tokens = $14,000 a day; 4M x 300 = 1.2B output tokens = $4,800 a day. About $18,800 a day, or roughly $560,000 a month.
- Design consequences: generate the suggestion only when the agent clicks (not on every ticket open), cache the stable system prompt with provider prompt caching, trim ticket history, and route simple tickets to a smaller, cheaper model. Each of these is a lesson in this module.
Picture it
Calling an LLM is like asking a brilliant but slow, expensive outside consultant who sometimes misreads the brief and will follow instructions hidden in any document you hand them. You would not stand at their desk waiting for every question, you would not give them the keys to the bank, and you would check their work before sending it to a customer.
Key patterns
LLM as an unreliable remote dependency
When: Any synchronous model call
How: Set a timeout, retry only on 429/5xx with backoff and jitter, use a circuit breaker and a fallback model, and define what the user sees when everything fails.
Stream when a human waits, queue when nobody does
When: Choosing how to call the model
How: Interactive chat streams tokens over SSE. Bulk summaries, document ingestion and long agent runs go to a queue or durable workflow and notify on completion.
Cheapest path first
When: High-volume features
How: Handle common, structured requests with rules, search or a small model; send only the hard remainder to a large model. Cost and latency both fall.
Validate model output like user input
When: Model output drives UI, code or actions
How: Request JSON against a schema, validate types and allowed values, reject or repair invalid output, and never run generated SQL or commands with broad privileges.
Real-world use
Klarna's AI customer service assistant
Klarna said in early 2024 that its assistant handled about two thirds of customer service chats in its first month (around 2.3 million conversations). A workload like that is only practical with a gateway, strict tool permissions and a hand-off to human agents when the model is not confident.
Uber's GenAI Gateway
Uber has written about an internal gateway that all teams use to reach multiple LLM providers, with central authentication, logging, cost tracking and PII redaction. It is the 'one door for every model call' pattern applied at large company scale.
DoorDash support chatbot for Dashers
DoorDash described a RAG-based chatbot that answers delivery partners' questions, with an LLM guardrail that checks answers before they are shown and an LLM judge that scores quality offline. Quality checks are part of the architecture, not an afterthought.
ChatGPT and provider APIs streaming over SSE
OpenAI and Anthropic stream responses as Server-Sent Events, token chunk by token chunk. Products built on them stream to the browser too, because a 10-second answer that starts in under a second feels fast.
In a project
A Bengaluru food delivery company wants an 'Ask about your order' chat in its app. It must answer questions like 'where is my order' and 'why was I charged a packaging fee' in English, Hindi and Hinglish, during dinner peaks of 30,000 chats a minute.
- Answer the easy, high-volume intents (order status, ETA) without an LLM: a small intent classifier plus a template reading the order service. Only open-ended questions go to the LLM.
- Route every LLM call through the company's AI gateway with a per-feature config: primary model, cheaper fallback model, max 400 output tokens, 15-second timeout.
- Stream answers over SSE so users on slow 4G networks see text within about a second; send a heartbeat every 15 s so mobile proxies do not cut the connection.
- Give the model read-only tools (get_order, get_bill_breakdown) that use the customer's own identity, never a service-wide admin key. Refunds go through a separate API that applies business rules and limits.
- Put the order data into the prompt as structured fields, mark user text clearly as untrusted, and validate the model's JSON answer before rendering.
- If the gateway reports both providers failing, show the normal order-tracking screen and a 'chat with support' button instead of an error.
- Log a trace per chat (prompt, tool calls, tokens, cost) with phone numbers and addresses redacted, and keep a 300-question Hindi/English evaluation set that must pass before any prompt change ships.
Impact: About 70% of chats are answered by the cheap non-LLM path, LLM spend stays within the monthly budget even on match nights, a provider outage turns into a quiet fallback instead of an incident, and no prompt injection can trigger a refund because the model never holds that permission.
Trade-offs
Use it when
- The task needs language understanding or generation that rules cannot do well: summarising, drafting, classifying messy text, answering questions over documents
- A few seconds of latency is acceptable, or the work can run in the background
- The value per call clearly exceeds the token cost, for example deflecting a support ticket that costs a human agent several minutes
Avoid it when
- A deterministic rule, SQL query or classic ML model solves it: order status, tax calculation, eligibility checks
- The answer must be exactly correct every time with no human review, such as moving money or legal decisions
- The path has a tight latency budget (under about 200 ms) and cannot be precomputed or cached
- You cannot send the data to an outside provider (for example regulated payment data under RBI localisation rules) and have no approved self-hosted model
Common mistakes
- Drawing the LLM call as a fast synchronous hop and sizing servers for 50 ms requests, so the system runs out of connections at modest traffic.
- Estimating only requests per second and never tokens, so cost and provider rate limits are discovered in production.
- Letting every service call providers directly with its own API key: no central budget, no failover, no single place to redact data or switch models.
- Trusting the system prompt ('never reveal other customers' data') as a security control instead of limiting what data and tools the model can reach.
- Having no degraded mode: when the provider returns 503, the whole page errors instead of falling back to a non-AI experience.
- Shipping prompt or model changes with no evaluation set, so quality regressions are found by customers.
Interview questions
Quick revision
- LLM latency: ~0.3-2 s to first token, 30-150 tokens/s after; long answers take 5-60 s.
- In flight = rate x latency. 100/s x 10 s = 1,000 concurrent requests.
- Cost = calls x (input tokens x input price + output tokens x output price). Output tokens usually cost several times more.
- Provider limits are per minute in requests (RPM) and tokens (TPM); plan quotas like capacity.
- English is roughly 4 characters per token; Indic scripts often need several times more tokens.
- Stream when a person waits; queue or workflow when nobody waits or it takes > ~30 s.
- One gateway for all model calls: keys, routing, failover, budgets, cache, redaction, usage logs.
- Model output is untrusted: schema-validate it, never execute it with broad privileges.
- Prompt injection cannot be prevented, only contained with least privilege.
- Always define the degraded mode when every model is down.
- Every prompt or model change runs an evaluation set first.
Check yourself
1. A feature receives 50 LLM requests per second and each takes 12 seconds to finish. Roughly how many requests are in flight?
2. Which statement about prompt injection is most accurate?
3. Which workload should NOT call an LLM synchronously on the request path?
4. Why is estimating only requests per second not enough for an LLM feature?
See it in a full design
These walkthroughs use this lesson.
Saved on this device. Create a free account to keep it on every device.
Practise this with Sam
A real mock interview, out loud, with an honest scorecard. Free to start.