Warm up, then benchmark the workflow

Tokens per second is the wrong number to lead with. Generation speed is real and it matters, but it only covers what happens after the part everyone actually sits through waiting.

A single call on your machine is at least four clocks stacked together: the model loads off disk, the kernels spin up, the prompt gets chewed through, and only then does steady decoding start. Tune the last clock alone and you can double a figure a real session never runs long enough to notice.

Write down what you're running before you touch a setting: interactive chat, code completion nudging you mid-keystroke, document extraction in a queue, or an overnight batch job. The machine underneath doesn't need to be exotic: an ordinary workstation you already use tells you more than a bench rig, because you'll actually live with its queueing and thermals.

The clock starts before the model does

Pick a fixed prompt set and don't swap it mid-comparison. Log cold start, warm time to first token, prompt speed, generation speed, and energy draw if you can measure it, and keep the launch command with each result. One warm sample proves nothing. Run a cold start whenever a human will hit one, and repeat until heat or queueing shows up. Watch the output too, not just the clock: if the answer changes when the speed does, you've swapped systems, not tuned one.

Record time to first token, prompt speed, generation speed, peak memory, wall power when you have a meter, and whether the task passed. Report the median for the normal case, but watch the slow end too, that's the stall a person swears at, not the average. A results file that fits on one screen looks like this:

task:       one fixed, named workload
prompts:    same set, versioned
quality:    pass / fail / abstain
timing:     cold, warm, p95
resources:  memory, power, disk
verdict:    keep, revert, retest

Fan noise, responsiveness, and setup friction belong right there too, none of it rigorous. It still decides whether the setup survives past next week.

Five percent wins don't survive an upgrade

The failure mode I see most is publishing the fastest tokens-per-second figure and skipping the wait before it. 'It ran without crashing' isn't a performance result, and 'the output read fine to me' isn't an evaluation. Check runtime logs and the operating system's own metrics instead of trusting whichever flag you passed. When two runs disagree, change exactly one variable and say what mechanism you expect to move, or the benchmark is a pile of anecdotes with timestamps.

A fragile five-percent win also won't survive the next model, driver, or runtime update, so don't build a monitoring platform to defend it. A script you run without thinking, pointed at the same prompts, dumping into a plain text file, is enough to retest after every upgrade.

My rule: benchmark the whole sequence a person sits through, idle gaps and model swaps included, not just the part after everything's warm. Try the smallest plausible change first. Leave headroom on purpose instead of tuning to the hardware's edge, that slack absorbs the longer prompt you didn't plan for and whatever you'll add later.

Here's the trade I make on purpose: I give up the biggest tokens-per-second number, for a setup whose behavior I actually understand, one that still tells the truth after the next driver update instead of quietly drifting from what I measured. That's a smaller number on the page. I'll take it.