ConceptIntermediate 22 min· Shipping AI to production · 19 of 19

Production: caching, cost control and monitoring

Turn an AI demo into something you can run for real users: cache answers, cap tokens and budgets, add retries and fallbacks, and monitor cost, latency and quality without leaking personal data.

A demo that works on your laptop is not a product. In production you have many users, a bill that grows with every token, a model that sometimes fails or answers slowly, and no idea what is happening unless you measure it. This lesson covers the three habits that make an AI feature safe to run: cache what you can, control cost, and monitor everything.

It assumes you know tokens and pricing from *Tokens, cost, latency and choosing a model*.

Where the money and time go

LLM APIs usually charge per token, with output tokens costing more than input tokens at most providers (check your provider's price page). Time works the same way: latency grows with the number of tokens generated. So every saving comes from sending fewer tokens, generating fewer tokens, using a cheaper model, or not calling the model at all.

Lever	What you do	Trade-off
Call less	Cache answers to repeated questions	Stale answers if data changes
Send less	Shorter prompts, only the top retrieved chunks, trimmed chat history	Too little context hurts quality
Generate less	Set `max_output_tokens`, ask for concise answers	Cut-off answers if set too low
Use a cheaper model	Route easy tasks to a small model, hard ones to a large one	Needs evals to prove the quality holds
Limit use	Per-user daily budgets and rate limits	Some users hit limits

Caching

There are three kinds, and they stack.

Cache	What it does	Notes
Exact-match response cache	Same input gives the stored answer, no model call	Normalise the input (trim, lower-case). Best for FAQs. Never cache answers that depend on a user's private data under a shared key.
Provider prompt caching	The provider reuses work for a long, repeated start of your prompt (system prompt, documents) at a lower price	Put the stable part first and the changing part last. Rules, minimum sizes and discounts differ per provider, so read their docs.
Semantic cache	Uses embeddings to find a similar earlier question and returns its answer	Saves more calls, but can return a wrong answer for a question that only looks similar. Use a strict similarity threshold and test it.

Cache keys and privacy

A cache key must include everything that changes the answer: the model, the system prompt version, the user's identity if answers are personalised, and the question. A cache that mixes up users leaks private data.

Caching with Redis, including expiry (TTL), is covered in *Caching with Redis* in Backend Development. For LLM answers, set an expiry so that stale answers go away.

A wrapper with cache, retries and logging

Put every model call behind one function so that caching, limits, retries and logging live in one place. This example uses Gemini and a plain Python dictionary as the cache. Install with pip install google-genai and set GEMINI_API_KEY.

llm.py

import hashlib
import json
import logging
import time

from google import genai
from google.genai import types

logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("llm")

client = genai.Client()  # reads GEMINI_API_KEY from the environment
MODEL = "gemini-flash-latest"  # use a current model name from Google AI Studio

_cache: dict[str, str] = {}  # use Redis in production so all servers share it

def _key(system: str, question: str) -> str:
    raw = json.dumps([MODEL, system, question.strip().lower()])
    return hashlib.sha256(raw.encode()).hexdigest()

def ask(system: str, question: str, max_tokens: int = 200) -> str:
    key = _key(system, question)
    if key in _cache:
        log.info(json.dumps({"event": "llm_call", "cache": "hit"}))
        return _cache[key]

    last_error = None
    for attempt in range(3):                     # retry transient failures
        started = time.perf_counter()
        try:
            response = client.models.generate_content(
                model=MODEL,
                contents=question,
                config=types.GenerateContentConfig(
                    system_instruction=system,
                    max_output_tokens=max_tokens,  # hard cap on answer length
                    temperature=0,
                ),
            )
        except Exception as err:                 # in real code catch the SDK's specific errors
            last_error = err
            log.info(json.dumps({"event": "llm_error", "attempt": attempt, "error": str(err)[:200]}))
            time.sleep(2 ** attempt)             # wait 1s, 2s, 4s
            continue

        usage = response.usage_metadata
        log.info(json.dumps({
            "event": "llm_call",
            "cache": "miss",
            "model": MODEL,
            "input_tokens": usage.prompt_token_count,
            "output_tokens": usage.candidates_token_count,
            "latency_ms": round((time.perf_counter() - started) * 1000),
        }))
        _cache[key] = response.text
        return response.text

    raise RuntimeError(f"LLM call failed after 3 attempts: {last_error}")

if __name__ == "__main__":
    system = "You are a Brew & Bean assistant. Answer in under 20 words."
    print(ask(system, "What time do you open?"))
    print(ask(system, "what time do you open?"))   # same after normalising: cache hit

Run python llm.py. The second call is served from the cache. Each real call logs one JSON line with token counts and latency, which is the raw material for monitoring.

Example output (numbers will differ)

{"event": "llm_call", "cache": "miss", "model": "gemini-flash-latest", "input_tokens": 24, "output_tokens": 12, "latency_ms": 840}
We open at 8 am every day.
{"event": "llm_call", "cache": "hit"}
We open at 8 am every day.

Cost control

Cost control steps

1Set max_output_tokens on every call. Without it, one odd request can produce a very long, costly answer.
2Trim the input: cap user message length, keep only the last few chat turns or a summary, and retrieve fewer, better chunks in RAG.
3Route by difficulty: a small, cheap model for classification and simple answers; a stronger one only when needed. Prove it with your evals.
4Add a per-user budget and rate limit, as below, so one user or a script cannot burn your whole quota.
5Set a spending alert or hard budget in your cloud billing console. Use the free tier's rate limits as natural protection while learning.
6Look at your logs weekly: which feature, user or prompt uses the most tokens? Optimise that first.

budget.py

import datetime

DAILY_TOKEN_LIMIT_PER_USER = 20_000
_usage: dict[tuple[str, datetime.date], int] = {}

def allow(user_id: str, tokens_wanted: int) -> bool:
    today = (user_id, datetime.date.today())
    used = _usage.get(today, 0)
    return used + tokens_wanted <= DAILY_TOKEN_LIMIT_PER_USER

def record(user_id: str, tokens_used: int) -> None:
    today = (user_id, datetime.date.today())
    _usage[today] = _usage.get(today, 0) + tokens_used

Call allow(user_id, estimated_tokens) before the model call and record(user_id, actual_tokens) after. In production store the counters in Redis or a database.

Protect your key

An API key in a mobile app or in front-end code can be copied by anyone, and the bill will be yours. Keep keys on your server and call the model from there, behind login and rate limits.

Reliability

Reliability checklist

Timeouts: never wait forever for a model. Set a client timeout and show a friendly message.
Retries with exponential backoff for rate-limit (429) and temporary server errors (5xx), but not for bad requests (4xx).
A fallback: a second model, a cached answer or a polite "please try again" if the provider is down.
Streaming for chat UIs, so users see text early even when the full answer is slow (see *Streaming responses to a UI*).
Validate structured output and retry once if it is invalid.
Pin the model version where the provider allows it, and re-run your evals before upgrading, because behaviour changes between versions.

Monitoring

Traditional monitoring (errors, CPU, response time) still applies, but AI apps need more, because a model can fail without any error: it just gives a bad answer. Track these:

Signal	Why
Tokens in and out, cost per request and per user	Spot cost spikes and expensive features
Latency (p50 and p95, plus time to first token)	The slowest 5% of requests is what users complain about
Error and retry rates, rate-limit hits	Early warning of provider or quota trouble
Cache hit rate	Shows if caching saves money
User feedback (thumbs up or down) and follow-up rephrasing	The closest real signal of quality
Sampled conversations with the prompt, retrieved chunks and answer	Lets you debug bad answers and find new eval cases
Safety events: blocked inputs, failed validations, refusals	Shows attack attempts and guardrail gaps (see Guardrails, safety and prompt-injection defence)

Start with structured JSON logs like the ones above, sent to wherever your app logs go (see *Monitoring, logs and alerts* in Cloud and DevOps). When you outgrow that, dedicated LLM observability tools exist, some open source, that record traces of each step (retrieval, prompt, tool calls, answer). Compare current options before choosing one.

Logs are personal data

Logs of prompts contain what users typed, which may include phone numbers, health details or passwords. Mask personal data before logging, restrict who can read logs, and delete them after a set time. Mention this in your privacy policy.

The most valuable habit: when you find a bad answer in production, add it to your eval set, fix it, and run the evals before shipping. Monitoring finds problems; evals stop them coming back.

How this comes up in interviews

System-design questions like "Design a chatbot for 1 lakh users" are often judged on this lesson. Cover: API gateway with auth and rate limits, a cache layer, per-user budgets, model routing, streaming, retries and fallbacks, structured logs with token and latency metrics, feedback collection, evals in CI, and privacy of logs. Then give a back-of-envelope cost estimate: requests per day times average tokens times your provider's price per token.

Try it: Add budgets and cost to the wrapper

Extend llm.py so that: (1) ask takes a user_id, calls allow before the model call (raise an error if over budget) and record afterwards using input plus output tokens; (2) each log line includes an estimated cost computed from two constants you set from your provider's current price page; (3) a function summary() prints total calls, cache hit rate and total tokens from the log lines you collected in a list.

Key takeaways

Production AI needs three habits: cache, control cost and monitor.
Savings come from fewer calls, fewer tokens in, fewer tokens out, or a cheaper model; prove quality with evals.
Cache keys must include model, prompt version and user identity where answers differ; set an expiry.
Always set output token limits, per-user budgets, timeouts, retries with backoff and a fallback.
Log tokens, latency, errors, cache hits, feedback and sampled conversations, while masking personal data.
Turn production failures into eval cases.

Check yourself

Finishing the quiz marks the lesson as done.

1. Which is the safest cache key for a personalised answer?

2. Why set max_output_tokens on every call?

3. An AI answer is wrong but your logs show HTTP 200 and no errors. What does this tell you?

4. When should an API call be retried automatically?

5. What is the main risk of a semantic cache?

Saved on this device. Create a free account to keep it on every device.

Prompting vs RAG vs fine-tuning

Practise this with Sam

A real mock interview, out loud, with an honest scorecard. Free to start.

Start free