Measure tokens per joule, not only tokens per second
So you bump the batch size, the tokens-per-second number climbs, and everyone nods like the problem's solved. Nobody checks what the wall pulled to get there.
That's the trap on any machine you run day after day, not a bench for a screenshot. My reference box does nightly summarization and daytime chat, nothing exotic, the class of setup I lay out in hardware for local LLMs. No lab conditions, no claim this rig speaks for every machine out there. Peak throughput is cheap to chase, and it hides the number that decides whether a setup survives: energy per unit of finished work, on hardware with a power budget and a thermal ceiling that does not care how good your screenshot looks.
what the wall actually pays
Write the job down before touching a setting: interactive chat, code completion, document extraction, an overnight batch, whatever it is. Measure wall energy for that workload after warmup, and report quality beside the joules. Same inputs every run, launch command saved with the result. One warm sample tells you nothing. Include a cold start if a human will hit one, run long enough to expose heat and queueing: a faster wrong answer is not a win.
A speedup that changes the answer is a different system, not a faster version of the old one.
Record only numbers that could change the decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power, task success. Medians describe the normal case. A slow percentile catches the pause an average quietly erases. Note the friction too, because that decides whether a setup survives a real week of work, not the chart. An unattended run like the one I covered in the overnight batch piece lives or dies on queueing, not the peak number.
the number that survives a driver update
The failure mode I keep hitting: comparing board power limits pulled from a software sensor as if that number described the whole system. Local inference is full of configurations that technically run. Loaded is not a result. Looked fine is not an evaluation. Check the runtime logs and the OS's power and memory metering instead of trusting a flag you asked the driver for, since it will happily agree with a request it silently ignored. Change one variable per run and say what mechanism you expect to move, or the log is just anecdotes with timestamps.
There is a maintenance cost. A fragile five-percent win does not survive the next model, driver, or runtime bump. Keep the harness small: one short script, a handful of prompts that represent what you do, a plain-text results file you can diff. Enough to retest after every upgrade without standing up a monitoring platform for one workstation. Nobody needs that.
Test the smallest change first. Spare headroom absorbs the longer prompt, the background job, whatever model you try next month. Stop tuning once the workflow clears its latency and quality bar, and walk away. Workload boundaries and fewer wasted tokens are usually the first real win, before the hardware.
Optimize for finished work per joule. Everything else is decoration.