Prefix caching is the easiest local speedup to miss

My local inference server logs a cache-hit percentage on every request, sitting right next to the latency number, and for a long stretch I never looked at it. Most of what I run through it shares a spine: the same system prompt, the same schema, the same repository summary pasted at the top, call after call. Same shape, every time.

The rig is an OpenAI-compatible local server with prefix-cache support, nothing exotic, no bench-only tuning, no claim that any of this maps cleanly onto whatever you're running. What I care about is whether the feature moves a number that matters on hardware I actually own and have to keep running tomorrow, not whether it looks good in a five-minute demo. So before touching any setting I write down the job first, interactive chat, code completion, document extraction, or an overnight batch, and I decide what counts as a win before I start changing things.

The prefix has to stay a prefix

The method is almost insultingly simple: put the stable text first, normalize how you serialize it so the bytes match call to call, and watch cache-hit rate move alongside latency instead of watching latency by itself. Use identical inputs on every run and keep the exact launch command next to the result, because six months from now you will not remember which flags you had set.

One warm run tells you almost nothing. Run a cold start if your users will ever hit one, keep the loop going long enough that thermal throttling or queueing gets a chance to show up, and log whether the output is still correct while you're at it. Change the output and you've swapped systems, not sped one up.

Only a handful of numbers are worth writing down after that:

• time to first token.
• prompt-processing speed and generation speed, tracked separately.
• peak memory, and wall power if that's a real constraint for you.
• task success, meaning it solved the request rather than merely produced text.

Take medians for the normal case, but keep an eye on the slow end too. A p95 catches the pause that makes a workflow feel broken even when the average looks fine; a median alone will hide that every time. Whatever friction the setup adds, flakiness, an extra manual step, a config file nobody remembers editing, write it down beside the numbers, because that friction is what actually decides whether anyone keeps using the thing. If you want the deeper mechanics of how the underlying cache gets reused, prompt caching, the deep version covers that layer; this is about what you'd change in your own setup, not how the engine works internally.

A timestamp at the top will undo all of this

The failure I run into most often is someone putting a timestamp or a request ID at the very top of the prompt, ahead of the system instructions, then wondering why the cache never warms up. That one line invalidates the entire prefix on every call. Local setups are full of configurations that technically run without doing anything close to what you think they're doing. A model booting is not a performance result. A response that reads plausibly is not an evaluation either.

Check the runtime log and the OS-level metrics instead of trusting whatever flag you passed on the command line, because the flag tells you what you asked for, not what happened. When two runs come out different, change exactly one variable and say out loud what mechanism you expect to move. Otherwise the whole exercise turns into a pile of anecdotes with numbers taped to them.

There's a maintenance cost too, and it's a real one. A fragile five-percent win evaporates the next time the model, the driver, or the runtime gets bumped, and you won't notice until things feel slower and you can't say why. I wouldn't bother building a dashboard for this on a single workstation, that's overkill for the problem. A short script, a handful of prompts that represent what you actually run, and a plain-text file with the results: that's enough to retest after every upgrade without turning your desk into an observability team.

My rule, for what it's worth: shape the prompt for cache identity before you reach for more hardware. Try the smallest change that could plausibly work first, leave yourself headroom, and stop tuning the moment the workflow clears your latency and quality bar. Spare capacity is what absorbs the longer prompt next week, the background process you forgot was running, or the next model you want to try. If the hardware question is still live for you, hardware for local LLMs is worth reading before you spend anything.

Local models pay this kind of restraint back more than people expect. Clean workload boundaries and fewer wasted tokens usually beat another few gigabytes of memory, and the setup you end up trusting is the one whose behavior you actually understand, not the one with the flashiest single number on a chart.

So here's what I'd go check on your own setup before doing anything else: pull the runtime log for your last hundred or so requests, find the hit-rate column, and look at whatever sits before the system prompt. If there's a timestamp, a UUID, a session token, anything that changes on every call, move it to the end or cut it, then rerun those same requests and see if the number moves. That's the whole test, and it takes about five minutes.