Time to first token is a pipeline metric

A local chat UI on my laptop, pointed at a model server over the LAN, sits there for a beat before anything streams back. Nothing about the setup is exotic: same box that runs everything else, same home network, no lab conditions and no claim that one machine speaks for every machine. What matters is that the beat before the first token doesn't live in one place. It's spread across model load, a queue, tokenization, prompt ingestion, and whatever the network hop costs that day. None of that is decode speed.

Write the job down before you touch a setting. Interactive chat, code completion, document extraction, an overnight batch, each one tolerates that opening pause differently, and none of them get fixed by the same knob.

Four clocks hiding inside one pause

The number people quote when they call a model fast is tokens per second once it's already talking. That's the wrong number to stare at when the complaint is a sluggish start. Time to first token is really several things stacked end to end: the runtime finishing whatever it was doing before your request arrived, the request sitting in a queue, the prompt getting tokenized and pushed through the model as a forward pass that produces no output yet, and only then the first token leaving the process, followed by whatever the network adds on the way back to the UI. The mechanics of that last leg are the same ones covered under streaming latency architecture if you want the full shape of it. Speed up the decode loop all you like. If the model is still loading weights off disk or grinding through a long system prompt, the person watching the spinner does not care about your decode loop.

A five percent bump in tokens-per-second can ship and change nothing anyone notices, because the bottleneck was never sitting in that phase to begin with.

What actually gets stopwatched

So you time it properly. Timestamp the moment the request lands, the moment it clears the queue, the moment prompt processing finishes, and the moment the first token actually streams out. Use the same inputs every run and keep the launch command next to the result, because six months from now you will not remember which flag produced which number. One warm sample tells you almost nothing. Include a cold start whenever a human is going to hit one in practice, and repeat enough times to expose thermal throttling or queue backup that a single run hides completely.

What I actually keep in the result file:

• time to first token, both cold and warm
• prompt-processing speed and generation speed separately
• peak memory, and wall power when the setup runs on a meter I care about
• whether the task actually passed, not just whether the process exited

Medians describe an average Tuesday. A slow percentile describes the run that makes someone close the tab and go back to whatever they used before, and that's exactly what an average erases. Whatever friction or noise showed up during a run belongs next to the numbers too, because that friction is what decides whether the setup stays in use once the benchmark is over.

The kernel isn't always the knob

The trap I fall into, and watch other people fall into constantly, is tuning generation kernels while the model spends most of its wait loading or churning through a long prompt before decode even starts. Local inference has no shortage of configurations that technically run without doing anything useful. Loading counts as nothing on its own, no matter how relieved you are when it finishes, and looked fine is a vibe, not a benchmark result. Check the runtime logs and the operating-system metrics instead of trusting whatever flag you think you set, because the flag you requested and the flag that actually took effect are not guaranteed to be the same flag.

When two runs disagree, change exactly one variable and say out loud which mechanism you expect to move. Skip that step and the benchmark turns into a pile of anecdotes with numbers taped to them.

There's a cost here people underprice. A fragile five-percent win evaporates the moment a model, driver, or runtime updates, and you usually won't notice until someone asks why the thing feels slow again. A small script, a handful of representative prompts, and a plain-text result file are enough to retest after every upgrade. You do not need to stand up an observability platform to babysit one workstation.

Where I draw the stop line

My rule is boring on purpose: fix whichever pre-token phase is largest before you touch headline throughput at all. Test the smallest change that could plausibly move that phase, not the flashiest one available. Leave headroom instead of tuning right up to the edge, because that spare capacity is what absorbs a longer prompt, a background process you forgot was running, or the next model you'll want to try. Stop once the setup meets its latency and quality target; you are not obligated to keep tuning something that already does the job.

Hardware matters, and it's worth a look at what actually moves the needle in hardware for local LLMs before you spend money on any of it. But workload boundaries, controlled measurement, and cutting unnecessary tokens out of the prompt path tend to produce the first real improvement, well before a hardware swap does anything. The setup worth keeping is not the one with the single most impressive number in a spreadsheet. It's the one whose behavior you understand well enough to trust without checking every time.

Next time it feels slow, don't reach for a sampler setting first. Pull up the runtime log, find the timestamp where the request clears the queue, and see how much of the pause sits before that line versus after it. That split is what tells you which half of the pipeline actually owns the problem.