Session affinity reduces cache misses and creates failure domains

A KV cache is pinned to the GPU memory of the process that built it. It doesn't move to a second worker. That's the fact behind every unexplained slow second turn on a local inference box: the thing that makes it fast is stuck on one machine, and the load balancer has no idea it exists.

Round-robin and least-connections routing assume every request is interchangeable, fine for stateless HTTP, wrong for a multi-turn conversation. The second message needs the first message's cache. Send it to a different worker and the runtime recomputes the whole prompt from scratch. Nothing crashes. It just gets slower, quietly.

Picture two local inference workers behind a gateway. Before touching a launch flag, decide what number you're trying to move: first-token latency, completed jobs per hour, resident model count, energy per finished task, or fewer corrections a person makes afterward. "Faster" isn't a number, and won't tell you when to stop tuning.

Baseline first, opinions later

Pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings. Test against a small set of real-workload inputs, ugly cases included. Run once cold, storage untouched, model not yet loaded, if users hit that path. Then run warm, repeatedly, long enough for cache effects, queueing, memory pressure, and heat to surface.

The next experiment is sticky routing: key affinity by session, put a lifetime on it, and write down what happens when the worker holding that session disappears. That last part is the one people skip. A session with no defined recovery path isn't resilient. It's just undocumented.

Change one variable at a time unless you're deliberately comparing two whole configurations. Keep raw outputs from every run, not just summary numbers: quantization, context compression, and sampling changes can make a response faster and worse at once, so run the quality check against every candidate, not just the fastest one.

Where the time actually goes

A single "response time" figure hides everything worth knowing:

queue wait
model load or activation
prefill (prompt processing)
time to first token
decode rate / total completion
peak RAM, VRAM, power, swap
quality result: pass, retry, abstain, repair

Raw tokens per second is diagnostic, not the answer. For automation, count valid completed records per hour, not tokens. For coding, fold in review and correction time, since a fast wrong diff costs more than a slow right one. For chat, watch p50 and p95 first-token latency across realistic conversation lengths. A setup that wins a short batch-of-one benchmark can still lose the day once model swaps and long prompts show up.

The failure worth watching for is quiet: the gateway reroutes the next turn to a different worker, that worker eats a full prefill it never needed, and the only symptom is latency nobody can explain. Local runtimes keep going after the fast path is gone: offload layers, page memory, miss the cache, queue, fall back to a slower kernel, anything but fail loudly. Good for uptime, bad for anyone trusting a benchmark. Before believing a number, check:

• startup logs, for a silently chosen fallback path
• device placement, confirming layers landed where intended
• memory and swap counters during the run itself, not just after
• whether the optimization is active for your real tensor shapes and context length

Affinity is a discount, not a filing cabinet

Sticky routing is an optimization on top of conversation state, never the only copy of it. If the worker holding a session dies and the conversation can't be reconstructed elsewhere, what you built isn't a caching strategy. It's a single point of failure wearing a caching strategy's clothes.

Weigh that against operational cost, not just performance. A change that saves time but complicates startup, breaks observability, or blocks rebuilding the server after a disk failure is a bad deal. A small gain resting on an undocumented patch or manual cache re-warming is a poor trade beyond a single laptop. Boring configurations age well.

Write the conclusion down with the workload, date, and reasoning, next to the condition that should trigger a retest: new model family, driver update, longer contexts, different traffic mix. Numbers with no expiry date age into folklore that outlives the hardware they were measured on.

Leave headroom after hitting the target. Spare memory absorbs prompt-length variance. Spare queue capacity keeps a batch job from starving an interactive user. Thermal and power margin keep the box from throttling under sustained load. None of that is about full utilization. It's finishing the work predictably on the cheapest setup that clears the bar.

The real advantage of local inference is that every layer stays inspectable, worth remembering before hardware-for-local-llms tempts you to just buy more of it. What I'd check next: kill the worker holding an active session mid-conversation and watch whether the gateway recovers the state, or just quietly opens a new one and calls it the same chat.