Run the one-hour inference test

Plenty of people have shipped a local inference config after a two-minute smoke test, watched the token-per-second number look great, and only found out later that the cooler could not keep up with what they had asked it to do. Clocks step down, the fans hit their ceiling, and the fast setup from the demo quietly turns into a slower one nobody signed off on. By the time anyone notices, the job is already degraded, and the log that would explain why does not exist, because nobody thought to keep one.

None of this needs a lab. This is an ordinary local inference box, the kind of thing people put together for a homelab-llm-server: no chilled water, no acoustic foam. Write the job down before you touch a setting: interactive chat, code completion, document extraction, an overnight batch. "Faster" means something different for each of those.

Sixty minutes of the same prompt, on a loop

The method itself is not clever. Pick one representative prompt and loop it for an hour, logging clocks, temperatures, power draw, errors, and throughput as you go. Use identical inputs on every run, and save the exact launch command next to the result, because "I think I ran the same thing" isn't information. One warm sample tells you almost nothing. You want a cold start in the mix too, if a human will ever hit one, and the run needs to be long enough for heat soak or request queueing to show up. If the answer quality shifts along the way, what changed is the system, not just its speed, and that is a different decision than the one you thought you were making.

Nail those six things down before the first run. The numbers worth writing down are the ones that could move your decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power if you're metering it, and whether the task passed. Look at the median, then look past it to the slow percentile, the p95: that's the pause a person notices while the average quietly smooths it over. Note the fan noise and the setup friction too. Those costs get a configuration abandoned later on, long after the benchmark said it was fine.

Loaded is not fast

The trap I see most often is calling a build stable off a two-minute token-speed number. Local inference is full of configurations that technically run: the process starts, tokens come out, everyone moves on. Getting a response back doesn't tell you anything about speed. Getting a response that reads okay doesn't tell you anything about correctness. Check the runtime logs and the operating-system metrics instead of trusting whatever flag you passed on the command line; the flag tells you what you asked for, not what happened. When two runs disagree, change one variable and have a reason ready for why you expect it to matter. Skip that discipline and the benchmark stops being a benchmark, and turns into a pile of anecdotes with numbers stapled to them.

There's a maintenance cost here too, and it's easy to underweight. A fragile five-percent win from some clever flag combination evaporates the next time the model, the driver, or the runtime changes underneath you, and you won't notice until the job is already slower. I don't build a dashboard for a workstation. A small script, a handful of prompts that stand in for the real job, and a plain-text file of results is enough to retest after every upgrade, no observability platform required.

My rule: do not trust a machine with a long job until you've watched it behave for the length of that job. Test the smallest plausible change first. Leave yourself headroom on purpose. Stop tuning the moment the workflow clears its latency and quality bar, because that spare capacity is what absorbs the longer prompt, the background process you forgot was running, or the next model you decide to try.

Here is the trade I make on purpose: I give up the last few percent of throughput that a tightly wound, fragile config might squeeze out, in exchange for a machine whose behavior I can predict an hour into a job instead of two minutes into one. Chasing that last sliver of speed is real work, and it might be worth it for a one-off demo. For anything that runs unattended overnight, the kind of job covered in h100-overnight-batch, it is not, and I've stopped pretending otherwise.