Semantic caching needs a narrow blast radius

Semantic caching is one of the cheapest wins you can bolt onto an LLM system. It's also the fastest way to hand one tenant's answer to another tenant, or last month's policy to someone asking about today's, because the model still sounds confident while it's wrong.

I tested this against an internal assistant fielding the same handful of operational questions over and over. Nothing exotic, no claim that one setup speaks for every deployment. Before touching a threshold I wrote down what the job was (chat, code completion, document lookup, overnight batch) and what a good answer meant for it.

A cache key that knows who's asking

The trap is caching on embedding distance alone. Two questions can sit a hair's width apart in vector space and still carry different authorization scopes, or point at facts that changed since the first answer. A near-perfect cosine score says nothing about who's allowed to see the answer, or whether it's still true.

So the key carries more than the question text. I scope it by tenant and data version, exclude volatile intents (price, inventory, access, anything with a clock on it), store where each cached answer came from, and pick thresholds on the conservative side, not the side that flatters the hit-rate chart.

cache_key:   tenant_id + data_version + intent_class
excluded:    volatile intents (pricing, inventory, access)
threshold:   conservative, biased toward a miss over a wrong hit
provenance:  source query, timestamp, model version
ttl:         tied to the next data_version bump, not a clock

None of that is exotic. It's refusing to let a similarity score stand in for an authorization check, which is really a multi-tenant problem wearing a caching costume.

The measurements that would change my mind

Testing this means the same inputs every run, with the launch command written down next to the result. One warm sample proves nothing. I include a cold start whenever a human will actually hit one, run long enough to expose queueing or heat, and keep output quality beside the timing numbers, not off in a spreadsheet nobody reopens.

Record the numbers that could flip the decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power when it matters, task success. Medians describe the normal case; a slow percentile describes the pause that makes someone assume the system is broken. "It loaded" is not a performance result. "The answer looked fine" is not an evaluation. Check runtime logs and OS-level metrics instead of trusting a flag you passed, and change one variable per run, or the benchmark is an anecdote with a chart attached.

There's a maintenance tax too. A fragile five-percent win evaporates the next time a model or runtime gets bumped. A small script, a few prompts, and a plain-text result file are enough to recheck after every upgrade; anything heavier is effort on the wrong problem.

What I let cost money on purpose

The rule that's held up: cache stable, low-risk answers, recompute anything that's a decision rather than a lookup. Test the smallest plausible change first. Keep headroom, because that spare capacity absorbs a longer prompt, a background job, or next month's model. Stop tuning the moment the setup clears its latency and quality bar, not when the dashboard still has room to climb.

Here's the tradeoff, taken on purpose: a wider, looser cache would post a better hit-rate number and cost less to run. Mine won't win that comparison, and I'm fine with it. I'd rather give up some theoretical savings than explain to a tenant why they saw someone else's data, or why the price they were quoted stopped being true an hour before they read it.