Caching LLM calls: the free lunch with a stale aftertaste

Response caching in LangChain is the rare optimization that really is three lines:

from langchain_core.globals import set_llm_cache
from langchain_community.cache import SQLiteCache
set_llm_cache(SQLiteCache(database_path="ci_cache.db"))

Identical prompt and parameters, and the answer comes off disk instead of the API. I turned this on for our CI in February and it's the best cost decision I've made this year. It also served stale output into a client deliverable for nine days in May. Both facts are true, so both go in this piece.

where it actually paid

Two places, in descending order of surprise. Our CI suite has around 340 tests that touch a model, and most replay identical prompts on every push; after a week of warming, the hit rate settled somewhere near 70% and the per-run bill dropped from a real line item to a rounding error. The bigger surprise was the dev loop: re-running a client's 500-invoice extraction job after a downstream code change takes about four minutes from cache instead of the better part of an hour against the API, so people re-run it freely instead of debating whether they need to.

One distinction I keep having to make: this is response caching, which skips the call entirely. Provider-side prompt caching discounts the input tokens of a call you still make. They stack, and they fail differently.

the nine-day stale window

The incident was self-inflicted in an instructive way. For that extraction job I decided the framework's exact-match key was too conservative, since cosmetic prompt edits would invalidate a warm cache. So I wrote a clever wrapper keyed on the document hash alone.

Then a non-cosmetic change shipped: the prompt gained a field for payment terms. Document hashes didn't move, the cache kept answering from the old prompt, and the new column stayed empty for nine days until the client's ops person asked when payment terms were coming.

Backfill, an apology call, and a Tuesday evening I'd like back.

A cache key is a claim about what can't matter, and mine claimed the prompt couldn't matter.

I deleted the wrapper that week. The boring exact-match key would have missed and refetched, correctly, at the price of some redundant calls. I had optimized away the framework's one honest guarantee.

semantic caching, politely, no

Semantic caches key on embedding similarity: a question close enough to a cached one returns the cached answer, and the pitch is a much higher hit rate. The catch is that similar question and same answer are different claims. Whether a policy covers flooding and whether it excludes flooding sit nearly on top of each other in embedding space, and a threshold loose enough to catch paraphrases will catch negations too. I allow semantic caching on suggestion surfaces where a near-miss is harmless, and nowhere facts are involved; the long version of that argument has the receipts.

Response caching is one layer of a larger cost stack, and it's the layer with the sharpest edge: the cheapest call is the one you never make, right up until it's a wrong call you keep re-serving.

Cache hard in CI, cautiously in production, and date your leftovers.