Give the KV cache an eviction policy

Evict a session's KV cache the moment the odds it comes back stop covering what the memory is worth to someone else. That's the whole policy. Everything else here just earns that sentence.

Cache residency is a service-level choice, not a switch you flip once and forget. It comes down to two numbers: how likely a session is to return, and how expensive reloading its context would be if it doesn't stay resident. The same tradeoff underlies prompt caching in general, just moved up from a single prompt to a whole conversation. Picture a shared local chat server, a handful of people on one box, sessions coming and going, nobody logging out cleanly. That's the workload here.

The rent a stale session still owes

Before touching a launch flag or adding a card, decide what actually has to improve. First-token latency. Accepted jobs per hour. Resident model capacity. Energy per completed task. Fewer corrections afterward. Pick one. Make it faster isn't specific enough: it won't tell you when you're done, or stop you from making things worse along the way.

The trap is pinning every session indefinitely because reuse feels cheaper than recomputing. Local runtimes are almost too cooperative about this: they'll offload layers, page memory around, eat a cache miss, queue a request behind others, or fall back to a slower generic kernel instead of failing. The server keeps answering either way. A badly tuned cache and a well tuned one look identical from outside, until the queue backs up on a busy afternoon.

So check. Read the startup logs. Confirm where tensors actually landed. Watch the OS's memory and swap counters, not what the runtime reports about itself. Confirm the optimization you asked for is running for the context lengths people actually send, not the short one you benchmarked with. Change one variable at a time, and keep the full output, not just the timing number: quantization, context tricks, and model swaps can all make an answer shorter while making it wrong.

What earns a slot and what gets kicked out

Pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings. Test against a handful of real inputs, including the annoying ones nobody wants to deal with. Run once cold if a real user will ever hit that path, then run warm long enough for queueing, memory pressure, and heat to show up, not just the first lucky sample.

Time the phases separately: queue wait, model load, prefill, time to first token, decode rate, peak memory and power, and whether the answer passed the quality gate or had to be retried. Raw token speed is a diagnostic, not the result. For a chat server that means p50 and p95 first-token latency at realistic conversation lengths, not a synthetic single request. A setup that wins a short warm test with one user can lose badly once real conversations, model swaps, and long contexts show up together, exactly what happens running vLLM for a team sharing one box.

Weigh the operational cost alongside the speed: what the change does to startup, to upgrades, to your ability to rebuild the server after a disk dies. A five percent win that depends on an undocumented patch, or someone manually keeping a session warm, isn't a gain worth keeping on a machine other people rely on. Leave headroom after hitting the target, too: spare memory for prompt variance, spare queue capacity so a batch job can't starve an interactive user, thermal margin so the thing survives a real afternoon, not a five-minute test.

Write the decision down with the workload, the date, and the reason, and write down what should trigger a retest: a new model family, a driver update, longer contexts, a second team sharing the box. Skip that and last quarter's numbers become policy nobody remembers agreeing to.

Keep a session's cache exactly as long as its odds of returning are worth more than what that memory could be doing for the next request in line. The instant they're not, evict it.