Exact, semantic and prompt caching

Exact caching stores a response under a hash of everything that affects it: model and version, prompt template version, system prompt, messages, tools, temperature and other parameters. A later request with the same hash gets the stored answer in a millisecond for free. It is always correct for deterministic use cases, but hit rates are low unless many users send exactly the same thing.

Semantic caching embeds the incoming prompt into a vector, searches for the nearest cached prompt, and returns that cached answer if the similarity is above a threshold, for example cosine similarity of 0.95. 'How do I cancel my order?' and 'how can i cancel an order' can share one answer. Hit rates can be much higher on FAQ-style traffic, but every hit is a guess: two prompts can be very similar in wording and different in meaning ('cancel my order' vs 'cancel my subscription').

Provider prompt caching (also called prefix caching) does not cache answers at all. The provider keeps the model's internal state (the attention key-value cache) for a prompt prefix it has just processed. If the next request starts with the same prefix, those input tokens are billed at a large discount and the first token arrives sooner. The answer is still freshly generated, so this is safe for every use case. Self-hosted servers such as vLLM offer the same idea as automatic prefix caching.

1. 1Decide per feature whether caching is allowed: answers that depend on the user's private data or on time ('what is my balance', 'what is on sale today') are not cacheable across users, or only with a very short TTL and a key that includes the user.
2. 2Exact cache key: SHA-256 over a canonical JSON of {tenant_id, model, prompt_version, system, messages, tools, temperature, top_p, max_tokens}. Store in Redis with a TTL (for example 24 hours) and record the hit in usage events.
3. 3Semantic cache lookup: embed the normalised user question (an embedding call takes roughly 10 to 50 ms), search a vector index filtered by tenant, model, feature and prompt version, take the nearest neighbour and compare its similarity to the feature's threshold.
4. 4Choose the threshold from data: build a labelled set of prompt pairs ('same answer OK' or 'not OK'), compute precision at several thresholds, and pick the one whose false-hit rate is acceptable for the feature, for example under 1% for customer support.
5. 5Invalidate correctly: bump prompt_version whenever the prompt template changes; for RAG answers, store the ids of source documents with each cache entry and delete entries when those documents change; use TTLs as a safety net.
6. 6Structure prompts for provider prompt caching: put static content first (system instructions, tool definitions, long reference documents, few-shot examples) and dynamic content last (user question, timestamps). Any change early in the prompt, even a date in the first line, breaks the prefix match for everything after it.
7. 7Know the provider rules: OpenAI caches prompts of at least 1,024 tokens automatically; Anthropic requires explicit cache_control breakpoints, charges a premium to write the cache and bills cache reads at about 10% of the normal input price; prefixes expire after minutes of inactivity unless a longer TTL is paid for. Check current documentation, since the numbers change.
8. 8Measure each layer separately: exact hit rate, semantic hit rate and sampled false-hit rate, and cached-input-token share from provider usage fields. Report money saved, not only hit percentages.