Paged attention is memory management, not magic speed

Better KV-cache allocation raises serving capacity while kernels and workload still determine latency.

Somebody on a shared inference box turns on paged attention, points it at the same single prompt they always use to sanity-check a build, and watches tokens-per-second barely move. They call the feature broken, or decide the whole vLLM story is marketing. Neither is right. That letdown has happened on enough teams to name directly: paged attention was never sold to make one conversation faster. It exists so GPU memory does not fragment when dozens of variable-length conversations sit resident at once, a different problem than a single-stream benchmark measures.

The constraint shows up on a shared GPU server carrying many active conversations of different lengths, where naive KV-cache allocation leaves gaps no incoming request quite fits. Fix that and more concurrent sequences pack into the same memory, raising admitted throughput under load. It does nothing for the one person waiting alone on a completion. So before touching a launch flag, write down which number you actually want to move: first-token latency, accepted jobs per hour, resident model capacity, energy per completed task, or fewer corrections downstream. "Make it faster" will not tell you when you are done.

Freeze everything before you flip the flag

Pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings, and write them down somewhere you will look again. Build a small set of test inputs from the real workload, including the awkward ones: long context, odd encodings, the request that always stalls. Run once cold, storage and model unloaded, if users hit that path in production, then run warm long enough to expose cache effects, queueing, memory pressure, and heat.

Time the phases separately instead of trusting one total:

Then measure admitted sequences, cache utilization, preemption, and tail latency before and after, not just the average. Tokens-per-second is a diagnostic, not the product result: for automation count valid completed records per hour, for coding count review and correction time, for chat watch p50 and p95 first-token latency across real conversation lengths. A setup that wins a short warm batch-one test can lose once model swaps and long prompts show up during the day.

The server keeps running even when it is wrong

Local runtimes stay stubbornly willing to serve, which hides a bad configuration. They will offload layers to the wrong device, quietly miss cache, queue behind a stuck request, or fall back to a generic kernel, and keep answering anyway. Nothing crashes to warn you. Read the startup logs, check device placement, and watch the OS counters while a request runs, then confirm the optimization is active for the tensor shapes and context length your workload actually uses.

Change one variable at a time unless you are deliberately comparing two complete configurations. Keep the outputs alongside the metrics: quantization, context compression, sampling tweaks, and model swaps can all change the answer while looking faster on a chart. Every candidate clears the same quality gate the baseline did. A shorter answer gets checked for completeness, not just timed; a faster extractor gets its output counted for validity, not for how many braces parsed.

The upgrade path is part of the benchmark

Operational cost sits beside raw performance and gets left out of most comparisons. Note what a change does to startup, upgrades, observability, rollback, and the ability to rebuild the server from a bare disk. A five-percent win that depends on an undocumented patch or someone manually warming the cache each morning is a bad trade for anything a team relies on. Boring, documented configurations tend to still be running next year.

Put an expiration date on the win

The rule that holds: value paged attention for predictable concurrency and memory efficiency under load, not for single-stream speed, and write that into the result file with the workload, the date, and the reasoning. Write down what should trigger a retest too: a new model family, a driver update, longer contexts, another user on the box, a different request mix. Skip that and old numbers quietly become folklore nobody can defend later.

Leave headroom once you clear the target instead of packing every resource to its limit. Free memory absorbs prompt-length variance you did not test for. Spare queue capacity keeps a batch job from starving the interactive user waiting at a terminal. Thermal and power margin is what lets the box run this way for months, and it is what leaves room to get more out of the local tier later instead of buying another card.

That inspectability is the real point of running local at all: every layer is something you can log and measure yourself, so performance never has to rest on somebody else's slide.

None of this changes the fact that if you are one person running one chat session on a single card at home, the concurrency story above buys you nothing: the headroom sits idle, and that phase table is more bookkeeping than a one-person setup will ever need.