Design a semantic cache for LLM calls

The biggest risk is not a low hit rate, it is serving an answer that was right for someone else. Three rules. One: never cache answers that depend on private or live state ('where is my order 123', 'what is my wallet balance'). The bot already knows which intents need tools or order data; only intents marked cacheable go to the cache. Two: everything that changes the answer must be in the scope, not in the similarity. The scope key is hash(tenant, feature, model, prompt_version, locale, user_segment if answers differ for Gold members). Similarity search runs only inside one scope, so a Hindi answer is never served for a Tamil question and tenant A never sees tenant B's answer. Three: answers built from documents carry the document versions they used, so a policy change can invalidate them precisely; TTL alone (say 24 h) is a backstop, not the main mechanism. Also never cache with temperature-driven variety if the product wants different phrasings, and do not cache refusals or error messages, or one transient failure gets served to thousands of users.

Cosine similarity is not a probability of 'same answer'. 'Can I cancel my order?' and 'Can I cancel my subscription?' can score 0.92 with general embedding models while needing different answers. So you measure. Build a labelled set of 2,000-5,000 question pairs from real logs, labelled 'same answer OK' or 'not OK' by support agents, including hard negatives that differ by one word (cancel/return, before/after delivery). For each threshold compute precision (of the matches served, how many were OK) and hit rate. Pick the threshold that keeps precision ≥ 99% for this feature; often that lands around 0.93-0.96, but it depends on the embedding model and domain, so never copy a number. Then run in shadow mode for a week: compute the would-be hit, still call the LLM, and have a judge compare the two answers on a sample. Only switch to serve mode when shadow precision matches the offline number. Keep a verifier band just below the threshold where a cheap model checks the pair; it raises hit rate safely at a small cost. Re-tune whenever you change the embedding model, because similarity scores are not comparable across models.

In a chat, the latest message alone is often meaningless: 'and for COD orders?' only makes sense with the previous turn. Embedding the raw last message gives false hits; embedding the whole conversation gives almost no hits. The usual fix is to cache on a standalone question: a small model rewrites the turn using recent history into 'how long does a refund take for cash-on-delivery orders?'. Many RAG bots already do this rewrite for retrieval, so the cache reuses it at no extra cost. Cache only turns where the rewrite is confident and self-contained. Normalise before embedding: trim, lowercase, remove greetings and filler ('hi', 'pls', 'bhai'), and transliterate common Hinglish spellings where your embedding model handles them poorly, since 'refund kab aayega' and 'when will refund come' should land close together. Use a multilingual embedding model if you serve Indian languages; English-only models scatter Hindi and Hinglish text. Keep the original user text out of the stored answer key, and store the rewritten question for debugging.

Whatever gets stored will be served to many users, so writes need more care than reads. An attacker can ask a crafted question that makes the model produce a misleading answer (for example via prompt injection asking it to include a phone number for 'refund help'), and if cached, that answer reaches thousands. Defences: only store answers that passed the output moderation and policy checks; never store answers from conversations flagged as suspicious; require an entry to be generated twice (two independent misses producing consistent answers) before it is served to others, or have it pass the verifier once; and cap how many users can be served a new entry in its first hour. Feedback closes the loop: a thumbs-down or a human agent correcting a cached answer quarantines the entry immediately and becomes a labelled negative for threshold tuning. Track false-hit rate per feature on a dashboard, and alert if quarantines spike, which often means a bad prompt version or a policy change that was not wired to invalidation.

Redis with vector search keeps everything in memory and gives single-digit ms lookups, which fits a latency-sensitive cache of tens of millions of short vectors. pgvector is simpler if Postgres already holds the entries and QPS is modest. A dedicated vector DB (Qdrant, Milvus) helps when the cache grows to hundreds of millions of entries or needs many tenants with heavy filtering. Pick Redis or pgvector here; the scale does not need more.

A large embedding model separates close-but-different questions better but adds latency and cost on every lookup, including misses. A small model (384 dims) is fast and cheap, and the verifier band covers its weak spots. Measure precision on your labelled set for both before deciding.

Serving directly saves the full LLM call. An alternative is to pass the cached answer to a small model to adapt it to the exact question, which cuts cost by maybe 70% instead of 100% but handles near misses better. Good for long answers; overkill for FAQs.

Adding prompt version, model and segment to the scope makes every deploy start with an empty cache. Warm the new scope by replaying the top 5,000 questions through the new prompt in the background before switching traffic, and avoid putting fields in the scope that do not actually change the answer.

Unbounded caching fills RAM with long-tail questions that never hit again. Evict by LFU or 'hits in last 7 days', keep TTLs, and keep only entries with at least one hit after a warm-up period. Most savings come from the top few thousand questions.

Lookups time out at 30 ms and the gateway calls the LLM as if it were a miss. Cost rises, users notice nothing. The cache is an optimisation and must never be on the failure path.

Feedback or the sampling judge flags it; the entry is quarantined and removed from the index within seconds. Because entries carry entry_id in logs, you can find every user who received it and, for serious cases, send a correction.

If the doc.updated event is missed, stale answers live until TTL. A nightly job compares source_doc_versions of active entries with current document versions and purges mismatches.