Benchmark your own box or believe strangers
In March I bought 64 GB of extra RAM for the 3090 box because of one stranger's number. A Reddit comment said a certain 70B ran at 8 to 9 tok/s split across GPU and CPU, which sounded livable for overnight batch work, so I ordered the DIMMs that same evening. The number turned out to be real. It also measured the one thing my workload doesn't do.
The commenter was chatting: a couple hundred tokens of prompt, long answers. My workload is the inverse, 10 to 14k tokens of code review payload in, three paragraphs out. Same model, same class of machine, different physics.
two numbers, and everyone quotes one
Local inference has two speeds, and they're barely related. Prompt processing (prefill) is how fast the model chews through your input; it's parallel and compute-hungry. Generation is how fast it emits new tokens, one at a time, gated by memory bandwidth. Short-prompt chat lives almost entirely in generation land. Long-prompt, short-answer work like review or retrieval lives in prefill land. The Reddit number was generation. My 70B's partially-offloaded prefill worked out to a minute and a half before the first token of every single review, and no generation speed rescues that.
Nobody posts the prefill number.
The RAM wasn't wasted; it caches models and absorbs the occasional huge context. But the purchase premise was wrong, and one question would have exposed it: at what prompt length?
context bends both numbers
The second thing strangers' numbers hide is context depth. Generation slows as the KV cache grows; my daily 14B opens around 45 tok/s on an empty context and sits in the low thirties by 20k tokens in. If your real sessions live at 15k of accumulated context, a hello-world benchmark measures a machine you don't actually own. Measure at the depth you work at, not the depth that flatters.
A tok/s figure without prompt size, context depth, and minutes-into-the-run attached is a rumor.
heat is a parameter too
On laptops the first minute lies. My MacBook's sustained generation sags by somewhere between a fifth and a quarter once the chassis is properly warm, ten-plus minutes into a real session. The full sad chart is in the thermals piece. The 3090 in a warm July office loses a little too, just less. So benchmark once cold, then again twenty minutes into actual load, and treat the second number as your life. And throw the very first run away entirely; model loading and cold caches contaminate it.
the nine-line habit
llama.cpp ships llama-bench, which prints both speeds separately, at whatever prompt sizes you name:
llama-bench -m qwen3-14b-q4_k_m.gguf -p 512,4096,12288 -n 128Mine runs from a nine-line shell script that tags each row with the date and runtime build, appends it all to a CSV, and gets invoked after every upgrade of anything: runtime, driver, macOS. Two runs, first one discarded. On Ollama the same two numbers appear under ollama run --verbose as prompt eval rate and eval rate; read both, and know that the server flags (batch size, flash attention) move prefill far more than they move generation.
Strangers' numbers still have a use: they tell you what's worth testing, never what's worth buying. It's the same reason the benchmark page on this site sticks to cloud models. Those numbers reproduce anywhere, while local throughput reproduces on exactly one machine in the world. Yours.