Idle power belongs in the GPU purchase decision

The box under the desk stays on pretty much around the clock for a scattering of real requests: a code completion before lunch, a chat question from someone on the home network that evening, and not much else worth the electricity.

The GPU never really works. It idles, and idling is not free.

Low utilization is the part everyone forgets to price in. A workstation that services intermittent requests, the kind of homelab box that answers a handful of pings a day, spends the overwhelming majority of its powered-on hours doing nothing generative at all: waiting, listening, holding a model resident in VRAM so the next request doesn't wait for a cold load. Buy hardware and pick launch flags around a five-minute benchmark run, and you've measured the least representative hour of that box's entire week.

Before you touch a launch flag or swap a card, decide what you're actually trying to move. First-token latency. Accepted jobs per hour. How many models you can keep resident at once. Energy spent per completed task, not per token generated. Fewer corrections from whoever reviews the output. "Make it faster" doesn't tell you when to stop, and it won't protect you from a change that speeds up the fast path while quietly breaking the slow one.

what a workstation actually does all day

Pin everything before you measure anything: model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Then run a small set of inputs pulled from the real workload, awkward cases included, not the clean demo prompt you'd show a colleague. Do one cold run with an unloaded model and cold storage if that's a path real users actually hit, then repeat warm runs long enough for cache effects, queue behavior, memory pressure, and thermal throttling to show themselves.

The run that tells you something meters sleep, idle, model-resident, and active states across a representative week, not a representative five minutes. Track phase-level timings instead of one lump total:

• queue wait
• model load or activation
• prompt processing and prefill
• time to first token
• decode rate and completion time
• peak RAM, VRAM, power draw, and swap
• quality pass, retry, abstain, or repair

Raw token speed is diagnostic, it isn't the product. For a batch pipeline, count valid completed records per hour. For a coding assistant, count review and correction time alongside generation time. For chat, look at p50 and p95 first-token latency against conversation lengths that resemble what people actually type, not one fresh prompt in an empty context. A configuration that wins a short warm single-request test can still lose the whole day once a model swap or a genuinely long context shows up.

a degraded server still answers your prompt

Here's the trap: comparing watts across a five-minute benchmark and calling the question answered. Local runtimes are stubbornly willing to keep working. They'll page memory, miss a cache, queue a request behind another, or drop to a slower kernel rather than refuse outright, and if you've worked through the offload math before you already know a runtime will push layers back to system RAM without so much as a warning. That willingness is great for uptime and terrible for measurement, because it hides a degraded configuration behind an answer that still looks fine on the screen.

Read the startup log. Check device placement. Watch the operating-system counters, not just whatever the runtime prints to its own console. Confirm the optimization you think is active is actually active for the tensor shapes and context length you're running, not the ones from the release notes. Change one variable at a time unless you're deliberately comparing two complete configurations against each other, and save the actual outputs alongside the metrics, because quantization, context compression, and sampling changes can all make an answer shorter or faster while making it worse. Run the quality gate on every candidate, every time. If a config produces a shorter answer, check that it's complete. If an extractor got faster, count valid records, not braces that merely parsed.

Operational cost sits right next to raw performance, not underneath it as an afterthought. Ask how a change affects startup, upgrades, observability, rollback, and your ability to rebuild the server from nothing after a disk dies. A five percent gain that depends on an undocumented patch or someone manually warming the cache each morning is a bad trade for anything more than one person relies on.

The rule that survives contact with a real week of usage is to manage the duty cycle, not the peak: sleep, wake-on-demand, or a deliberately smaller always-on tier sized to the actual request rate. Write the decision into the result file next to the workload, the date, and the reason, and write down what should trigger a retest too: a new model family, a driver update, longer contexts creeping in, a second user showing up, a different mix of traffic. Skip that second part and last quarter's numbers quietly turn into infrastructure folklore nobody bothers to double-check.

Leave headroom once you hit the target instead of packing the box to its limit. Free memory absorbs the prompt you didn't expect. Spare queue capacity keeps an interactive user from waiting behind someone else's batch job. Thermal and power margin is what lets the machine run for a week instead of an afternoon. None of that shows up as a win in a benchmark table, and all of it is the difference between a server and a demo that happened to work once.

All that said, wake-on-demand adds a cold-start tax that the one person actually using this box will notice and resent every single time, and a week-long metering run is a genuinely annoying amount of overhead to set up for hardware that mostly exists to answer a handful of household requests. Do the cheap version first. Measure properly only once idle power starts showing up somewhere it costs real money to ignore, like a shared power bill or a shared machine.