Measure quality-adjusted token speed

Rank configurations by accepted output per hour, not by raw tokens per second. That's the whole rule. Everything else here is what it takes to trust that number, because a model that spits out garbage fast isn't saving you time, it's just moving the delay to the part of the day when you're reviewing its output instead of waiting on it.

Pin everything before you touch a dial

Freeze the parts of the setup that shouldn't move before you compare anything: model file, tokenizer, prompt template, exact runtime build, launch command, sampling settings. Pull your test inputs from the real workload, awkward cases included, because that's where a fast-looking configuration usually comes apart. Run one pass cold, model unloaded and disk cache empty, if that's a path users actually hit, then run warm and keep running long enough for queueing, memory pressure, and heat to show up. Five minutes tells you nothing about an eight-hour shift. Track the whole timeline, not the headline figure: queue wait, load or activation time, prefill on your real prompt length, time to first token, decode rate, peak memory and power. None of that means much until you've named what "done" looks like for this job. An automation pipeline wants valid completed records per hour, retries counted in. A coding assistant's honest number folds in the minutes a person spends fixing what came out. A chat interface lives on p50 and p95 first-token latency once a conversation has some length, not on how snappy it looks empty. A configuration that wins a short warm run at batch size one can lose the whole day once model swaps and long prompts show up, and that's the gap a reference like llamacpp-server-flags is meant to close before you trust a launch flag in production.

Local runtimes are too forgiving for their own good

Local inference stacks are stubbornly willing to keep working. They offload layers to a slower device, page memory around, miss a cache, queue requests, or drop into a generic kernel rather than fail loudly, and that same resilience is what lets a degraded configuration post a plausible number. Read the startup log. Check device placement. Watch the operating system counters. Confirm the optimization you asked for is active for the tensor shapes and context length you're actually running, not the ones from someone else's demo. Change one variable at a time unless you're deliberately comparing two complete systems, and save the outputs, not just the timings: a shorter answer might be a truncated one, and a faster extractor might just be counting braces instead of valid records. Run the same quality gate against every candidate, the kind of check an evals-and-llm-as-judge setup does better than eyeballing the first three results. Weigh operational cost next to the speed number: what the change does to startup, upgrades, observability, rollback, and rebuilding the box after a disk failure. A five-percent gain that depends on an undocumented patch or someone manually keeping the model warm is a bad trade for anything shared. Boring configurations age well.

Where the number goes once you have it

Write the conclusion down: optimize accepted work per unit of time and energy, tied to the workload, the date, and the reason you believe it. Write the retest trigger too, a new model family, a driver update, longer contexts, another user on the box, a different traffic mix. Skip that and last quarter's benchmark quietly turns into folklore nobody questions. Leave headroom once you've hit the target instead of packing every resource to the ceiling: free memory absorbs the prompt you didn't test for, spare queue capacity keeps an interactive user from waiting behind a batch job, thermal margin keeps the box behaving the same at hour six as it did at minute one. A local setup means every layer of this is inspectable, so the speed claim doesn't have to come from a vendor's slide. Go pull the phase timings from whatever comparison you ran last, find the configuration that won on tokens per second, and rerun the quality gate against it using this week's real inputs, not last quarter's. If it still wins, write down why. If it doesn't, you've found the actual number.