Power-limit the GPU before buying more cooling

Decoding one token at a time is a memory-bandwidth problem, and a card's factory power target is tuned for a compute-bound one: rendering frames, not walking a KV cache through VRAM a single step at a time.

That mismatch is the whole reason this is worth doing before you shop for a bigger cooler. My reference box is a 24 GB NVIDIA card in an ordinary desk-side machine, no server room, nothing about the setup that's special. This rig isn't meant to stand in for yours. What matters is picking one job first, chat, code completion, document extraction, an overnight batch run, whatever it is. Write down what a good result looks like before you touch a single setting in the driver.

the watts that never make it into a token

During single-stream decode the compute units on the card spend a lot of their time waiting, not multiplying, because the next step needs weights and a growing KV cache pulled out of memory first. Feeding the GPU more power buys higher clocks, and higher clocks help a compute-bound job almost linearly. A memory-bound job doesn't care nearly as much. Past a certain point the extra watts stop turning into tokens per second and start turning into heat the fan has to move. Gaming firmware sets its power target for a workload that saturates the compute side of the chip. Local decoding, most of the time, does not.

turning the dial without kidding yourself

Finding your card's knee in the curve means sweeping power limits and logging four things every run: tokens per second, wall power, temperature, plus noise. Fix everything else. Same prompts, same launch command, saved next to the result so you can rerun it months later. Do a cold start if a human will ever hit one in production, and run each setting long enough for heat and queuing to show up.

workload:      one named, repeatable task
inputs:        fixed and versioned
quality gate:  pass / fail / abstain
latency:       cold, warm, p95
resources:     memory, power, disk
decision:      keep, revert, or retest

the six numbers that earn a spot in the log

Not every metric a monitoring tool spits out deserves to survive into your notes. Keep the ones that can actually change the decision:

• time to first token
• prompt-processing speed
• steady-state generation speed
• peak memory used
• wall power at the plug, when it's relevant
• whether the task itself still passes

Report medians, because they describe the run you'll actually live with, but keep a slow percentile too, since averages are good at hiding the pause that makes a tool feel broken. Write a line about noise and setup friction next to the numbers. A setup nobody wants to sit near gets abandoned regardless of what the log says.

how a five percent win turns into a liar

The trap is comparing peak throughput between two power limits and stopping there, missing that giving up a small amount of speed can remove a large slice of heat and fan noise. "It loaded" is not a result. "The answer looked fine" is not an evaluation, it's a vibe with a timestamp. Pull the actual number from the runtime log or the OS power meter, not the flag you passed on the command line, because a requested limit and an enforced one aren't always the same thing on every driver. Change one variable per run and say out loud why you expect it to move the number. A pile of runs where three things moved at once isn't a benchmark. It's an anecdote with axes.

There's a longer-tail cost too. A fragile win that only shows up under one specific model/driver/runtime combination stops existing the next time any of them updates, and you won't notice until the workflow just feels off. A short script and a handful of prompts that represent the real job, saved with a plain-text result file, are enough to catch that on the next upgrade. You don't need a dashboard for a single workstation.

the point where I stop touching the slider

My rule: keep the lowest power limit that still hits the interactive latency I actually need, and stop there rather than chasing the last few points of throughput. I test the smallest plausible cut first and leave real headroom under it. That spare capacity is what absorbs a longer prompt or a background job, and it's the same room I want free for the next model when I'm juggling more than one at once.

The tradeoff I'm making on purpose: I give up whatever throughput edge the factory power target has over my chosen limit. A few percent of peak tokens per second is worth less to me than a card that runs cooler and quieter, without a louder fan curve or a bigger heatsink standing in for a number I was never going to sit and wait for anyway. If your card's firmware won't take a power-limit command at all, which happens on some locked-down laptop parts, don't waste an evening reflashing vBIOS over this. Tune the software stack instead and leave the power table alone.