Count the full cost of local inference

A GPU that sits idle most of the day still draws standby power and still depreciates on the clock, whether or not it ever answers a prompt. That's the fact people skip when they weigh a card's price tag against a subscription invoice. What decides whether self-hosting pays off is utilization and the operational effort around it, not the sticker price of the box. A private chat used twice an hour and a nightly batch job can run on the same machine and still deserve completely different math.

The trap almost everyone reaches for first is dividing the purchase price by some theoretical lifetime token count and assuming continuous, perfect use. It's tidy, but it has no room in it for a stalled load or a Tuesday nobody touched the box at all. Before buying anything, or reconfiguring what you already own, the honest move is capturing a baseline, not shopping for a bigger card.

The record that has to survive a rerun

Capture the baseline with one named model, one fixed prompt set, and the exact server command you used to launch it. Keep the model artifact and the prompt template with it: those are the details that go missing and quietly invalidate a comparison a few weeks later.

Then annualize the hardware, meter the wall power, put a number on your maintenance time, and run the math twice: once for a low-utilization scenario, once for a high one. Don't average them. The two numbers tell you different things about the same box.

Measurements need to follow a request through the whole system, not just report a headline speed. Time to first token catches the loading and prompt work. Steady token rate describes decoding. Completion time is what the user experiences. Add peak memory and queue delay whenever they change the decision. If the workload is automated, count completed valid jobs per hour. If it's interactive, track the slow waits a person will remember.

I keep the log for this deliberately plain, one line per run:

• artifact, runtime, and launch flags
• workload and a fixed input set
• cold start, warm start, p50, p95
• peak memory and wall energy
• quality failures and abstentions
• decision, owner, retest date

A number with no decision attached is trivia. A setting with no owner attached is folklore.

Errors don't announce these failures

That price-per-token spreadsheet survives because the system keeps producing plausible-looking text even when something underneath it is quietly wrong. Local inference has failure modes that never throw an error: a model partly offloads to CPU, a cache misses without telling you, swap creeps up in the background, or a fallback path silently moves your privacy boundary.

Watch runtime logs and OS counters while the test runs, not after the fact. Change one variable at a time, unless the point of the run is comparing two complete configurations. Repeat enough to tell a durable improvement from a lucky sample. Read the outputs. Speed is not proof that two configurations produced equivalent answers, and treating it as proof is how a regression slips past a tokens-per-second benchmark. If a change makes an important task worse, write that cost down next to whatever it gained you.

Operational simplicity belongs in the ledger too. A clever optimization that needs a manual fix after every driver or model update isn't free, it's a recurring tax paid in attention. This is where the ponytail instinct earns its keep: prefer the boring configuration you can reproduce from a service file or a short script over the fragile one that only worked because you remembered the flags by hand. Keep the raw run artifacts out of the written report, but keep the hashes and commands, so a rerun months from now means the same thing it meant today.

Where the margin goes

The rule I'd carry forward is short: choose local for control, privacy, or a stable workload, then verify the economics with honest utilization numbers instead of aspirational ones. If you're already working through a local-first cascade for other tools, this cost accounting is the step that keeps the cascade honest instead of sentimental.

Visible limits and invisible limits behave differently. A visible one can be scheduled or priced into the next decision. An invisible one turns into unexplained waiting, then an emergency upgrade nobody budgeted for.

Stop tuning once the workload hits its quality and latency target with real headroom left over. That margin isn't wasted capacity, it's what absorbs a longer document, one more concurrent user, or whatever the next runtime release changes underneath you.

So the next thing worth doing isn't another benchmark run. Pull last week's utilization off the box itself, not off the spec sheet, and see which of your two scenarios it sits closer to. That single number tells you more about whether this was worth it than the price tag on the card ever will.