Stop giving llama.cpp every CPU thread
And that's the part nobody questions: the launch flag takes any number, so people type the biggest one nproc reports and call it done.
It runs. The first tokens even look quick. Then the prompt gets longer, a second terminal opens, and the whole thing bogs down in a way nobody can explain, because nobody wrote down what fast meant first.
where the extra threads actually go
Adding threads is not adding compute for free. Past some point they compete for the same memory channels and shared cache, and the contention costs more than the parallelism buys back. That's why the logical core count your OS reports is a fantasy number here, not a target: it counts hyperthreads sharing execution resources with the physical core next to them. Physical count is worth sweeping, logical count worth ignoring.
I run this on an ordinary desktop CPU handling prompt processing and partial offload, nothing exotic, no claim my numbers travel to your box. What travels is the method: name the job, chat or overnight batch, and decide what good looks like before you touch a flag.
a sweep you can trust later
Sweep physical-core counts, pin if the scheduler won't cooperate, and measure prompt processing and generation as separate phases: a gain in one can hide a loss in the other. Fix the inputs, and save the launch command with each result. One warm sample proves nothing. Include a cold start if a real person will hit one, and run long enough for throttling to show itself. Track time to first token, prompt speed, generation speed, peak memory, wall power where you can get it, and output correctness. Medians describe the usual case; the slow tail is the pause a user feels.
The setup worth keeping is not the one with the flashiest number, it's the one you understand well enough to leave running unattended.
why the good-looking number is often a lie
"It loaded" is not a benchmark, and "the reply read fine" is not an evaluation. Pull the runtime logs instead of trusting whatever count you requested; the scheduler doesn't always honor the flag you set. Change one variable per run and say why you expect it to move, or it's anecdotes with a timestamp. A five-percent win tied to this week's driver isn't a result, it's a coincidence with a due date. None of this needs a dashboard: a script and a text file are enough to retest after upgrades.
the rule and what it buys you
Find the plateau, then pick the smallest thread count sitting on it, not the single-run peak beside it. Stop tuning once latency and quality clear your bar. That headroom isn't wasted: it absorbs a longer prompt next week, or the next model on the same box. One limit worth naming: none of this replaces knowing the workload, and no sweep rescues a job you can't describe in a sentence.
Next time I touch this box, I'm not reopening the thread-count argument. I'm rerunning the sweep script against whatever changed upstream, and checking whether the plateau moved.