Average latency hides the local server you actually have

Averages are the wrong number to chase on a local inference box. That needs a qualifier right away: means aren't useless, they're just the wrong summary for a system where a human sits waiting for the first token. A workstation running mixed prompts, chat, completion, an overnight batch, can post a beautiful mean while still handing out a few ugly stalls nobody remembers to measure. The mean absorbs those stalls. It shouldn't be allowed to.

I test this on an ordinary shared workstation, nothing rack-mounted or climate-controlled, because that's the machine most people actually have. One box's numbers don't transfer to yours; the point is the method, not the number. Before touching a setting, write down what the job is: interactive chat, code completion, document extraction, batch. Decide what counts as a good result before you touch a lever, or you'll talk yourself into the number you wanted.

what the tail is actually telling you

Cold starts, model swaps, thermal throttling, a background process stealing cache, a queue behind a long context: same symptom, a handful of slow samples buried in a pile of normal ones. Average them in and they vanish. Keep percentiles by model, by context-length bucket, by warm or cold state, and the tail stops being noise and starts being a diagnosis. One warm run tells you nothing. Run long enough to let heat and queueing show, include a cold start whenever a real user would hit one, and log the launch command next to the result so you can tell, weeks later, what you ran.

What's worth recording is short: time to first token, prompt-processing speed, generation speed, peak memory, wall power if it matters, and whether the output was correct. A median describes the ordinary request. A p95 describes the request that made someone close the laptop in frustration. Keep both next to notes about noise and setup friction, because a fast config nobody wants to keep running is not a win.

workload:      one named, repeatable task
inputs:        fixed and versioned
quality gate:  pass / fail / abstain
latency:       cold, warm, and p95
resources:     memory, power, disk
decision:      keep, revert, or retest

the number that means nothing and the win that doesn't last

Averaging a cold run into a warm run produces a figure describing neither state, and a lot of local-inference tuning lives on that fiction. Loading successfully is not a benchmark result. An answer that merely looked fine is not an evaluation. Pull the runtime logs and OS metrics instead of trusting whatever flag you passed, because flags lie about what actually happened more often than you'd expect. Change one variable at a time and say why you expect it to move the number, or you're collecting anecdotes, not benchmarks. For the plumbing behind that first-token delay, streaming latency architecture is worth understanding before you blame the model.

There's a shelf-life problem too: a five-percent win from a fragile configuration evaporates the moment a driver, runtime, or model changes, which on a fast-moving local stack is often. You don't need an observability platform to catch that, just a small script, a few prompts, and a plain-text results file you can diff against last month's.

the part I have not sorted out

My rule: p95 finds the operational problem, median describes the day-to-day path, and once both meet the target, stop tuning and leave the headroom alone, because spare capacity absorbs the next long prompt or the next model you try. What I haven't sorted out is where the honest line sits between a single-GPU workstation like mine and an actual multi-request serving stack, the kind vLLM at home implies once you're serving more than yourself, where the tail gets shaped by other people's requests queuing behind yours, not your own thermal curve. I don't have a clean rule for when that's a shift in kind, not degree.