Caching LLM responses by meaning, and when that's a terrible idea
Worth clearing up a mix-up first: two very different things share the word "caching." Prompt caching is what the provider does to the input. It reuses the processing of a stable prefix, so you pay a fraction for the part of the prompt that didn't change. What I want to talk about here is caching the output: skipping the model call entirely and returning a stored response instead. That's a different mechanism, with a different payoff, and a much sharper way to fail.
The simplest version is an exact-match response cache: same request, byte for byte, gets the stored answer back. It's safe and trivial and almost useless, because LLM inputs are rarely identical twice. Two users asking the same thing phrase it differently, so exact matching treats them as two different requests. Your hit rate hovers near zero, and the cache earns you nothing.
Semantic caching is the interesting one. Instead of matching requests exactly, you embed the incoming query and check whether a similar enough previous query already sits in the cache. If one does, you return its stored response. Now "how do I reset my password" and "I forgot my password, help" can hit the same cached answer, because they mean the same thing even though they share few words. For the right workload this is huge. A support assistant or a docs Q&A bot fields the same handful of underlying questions thousands of times a day, and semantic caching can deflect a large share of them, cutting both latency and cost, because the cheapest model call is the one you never make.
The word doing all the work is "enough"
"Similar enough" is a threshold, and that threshold is a correctness dial wearing a performance costume. Set it tight and you cache conservatively: fewer hits, but high confidence the cached answer actually fits. Loosen it for a better hit rate and you start serving the stored answer to questions that were close but different in a way that mattered. "How do I reset my password on iOS" and "how do I reset the admin console password" can sit close in embedding space and have completely different answers. Cross that line and your cache returns the wrong answer quickly and confidently. That's worse than returning the right answer slowly, and worse in a way that's hard to spot, because the response still looks fine.
Staleness compounds it. A cached answer freezes at the moment you store it. If the underlying truth changes, the documentation gets updated, the policy gets revised, the price moves, the cache keeps serving the old reality regardless. This is bad enough with exact caching. It's worse with semantic, because you often can't even tell precisely which question a given cached entry is now answering for people.
And there's a context trap that bites hard in multi-tenant and personalized systems. If a response depended on the asking user's own data or permissions, it isn't safe to serve to a different user, no matter how similar their question looks. Semantically caching personalized answers across users isn't an optimization. It's a data leak waiting for the right pair of similar questions.
Where it fits and where it doesn't
The workloads semantic caching loves are high-volume and read-only, the kind where the same generic question shows up thousands of times a day, nobody's identity changes the answer, and an approximate match is fine. Think public documentation assistants, support deflection for common issues, anywhere the answer is the same for everyone and changes slowly. There, a conservatively tuned semantic cache is close to free money.
The workloads it ruins are the opposite. Anything personalized, where the answer depends on who's asking. Anything where freshness is the point. And anything where a near-miss answer does real harm: the medical, legal, or financial questions where "close enough" is exactly the failure you can't afford. For those, the safe similarity threshold is "exact," which is to say, don't bother.
If you do reach for it, build the safety rails in from the start. Keep the similarity threshold conservative, and resist the urge to loosen it for a prettier hit-rate dashboard. Scope the cache per tenant or per user wherever context matters. Put a TTL on entries so staleness has a ceiling. Give users a way to bypass the cache and force a fresh answer. Most importantly, watch the false-hit rate: the cases where a cached answer got served for a question it didn't actually fit. That number stays invisible until you measure it, and it's the number that tells you whether your savings came at the price of being wrong. Semantic caching trades correctness for speed and cost on a dial, and the whole job is keeping that dial honest.