The CPU-only local model is not a consolation prize

Fine, so the token stream stutters instead of pouring out instantly. That was never the metric I was optimizing for, and judging a CPU-only setup by how it feels next to a hosted chat product is how these boxes get dismissed before anyone checks what they were actually asked to do.

The machine I use for this is a recent desktop CPU, memory maxed out across both channels, no discrete GPU anywhere in the case. Deliberately unglamorous. It isn't a stand-in for every reader's rig, but it's ordinary enough that the numbers translate to yours, which is the whole reason I picked it over something exotic. If you want the fuller spec checklist, I laid it out separately in hardware for local LLMs.

Jobs per hour, not tokens per second

Before I touch a quant setting or a thread count I write the job down. Interactive chat, code completion, document extraction, an overnight batch. Each one has its own idea of what good looks like, and mixing them up is the quickest way to optimize a number nobody needed moved. A batch job doesn't care about a snappy first token. It cares about finishing before someone needs the output and getting every row right.

Once the job is named, the method is almost boring: pick a compact quant, push anything that can wait offline, keep the context window modest, and count completed jobs per hour instead of watching a token counter scroll. Same inputs every run, launch command saved next to the result, one cold start included, and the run repeated long enough that heat throttling or queue backlog shows up instead of hiding in a lucky first pass.

The numbers worth writing down

Not every number earns a place in the log. Time to first token, prompt-processing speed, generation speed, peak memory, wall power when it's relevant, and whether the task actually succeeded, that's the list. Medians tell you about the normal run; a slow percentile tells you about the run that made someone close the laptop in frustration, and averages quietly erase it. I keep a line for noise and setup friction beside the numbers too, because that's what decides whether a configuration survives past next week.

It loaded is not a result

Here's the trap: measuring CPU inference against how a hosted product feels. "It loaded" isn't a performance result. "The answer looked fine" isn't an evaluation, it's a shrug with extra steps. I check the runtime log and the OS-level memory and power counters instead of trusting whatever flag I asked the launcher to honor. The two disagree more often than anyone expects. Change exactly one variable between runs and be ready to say why you expect it to move the number. Otherwise what you've built is a folder of anecdotes with timestamps.

A win that doesn't survive the next driver update

There's a maintenance tax nobody puts in the spreadsheet. A fragile five-percent gain, squeezed out through some clever flag combination, evaporates the moment a model, driver, or runtime gets bumped. I don't build an observability stack for a single workstation. A small script, a handful of representative prompts, and a plain-text result file, that's the retest kit, and it's enough.

Headroom beats the leaderboard number

My rule: match throughput to the deadline, not to how fast someone can type. Test the smallest plausible change first. Leave headroom on the table on purpose. Stop tuning the moment the workflow clears its latency and quality bar, because that spare capacity is what absorbs the longer prompt, the background job you forgot was running, and whatever model you'll want to try next month.

The setup worth keeping isn't the one with the best isolated number. It's the one you understand well enough to trust unattended.

One limitation I won't paper over: none of this tells you whether CPU-only is the right branch at all, only how to measure it once you're standing in it. That routing decision is its own piece of work, and I've mapped it separately in the local-first cascade.

Next thing I'd actually check on this box: reboot cold, rerun the same batch job before anything else touches memory, and see whether prompt-processing speed lands inside its usual band. If it drifts, that's the signal to open the runtime log before touching a single setting again.