Memory channels matter for CPU LLM inference

Populate every memory channel before you spend another dollar on capacity. That's the position, and the qualifier is immediate: it only matters once the model is too big to live in cache, which for any CPU-hosted GGUF worth running, it always is.

Decode on a CPU spends most of its time waiting on RAM, not doing math. Every token needs the weight matrices to move from memory to the core again, and the core finishes its multiply long before the next chunk of weights arrives. Add cores and you get a faster queue for the same pipe. That's why the buying advice for hardware for local LLMs puts channel count at purchase time: it's a spec you check on the box, not a setting you fix later.

The slot count you didn't check

The mistake that keeps showing up: buying the biggest DIMMs a board will take and filling half the slots, because two 64GB sticks look like a cleaner shopping list than four 32GB ones. Same capacity, sometimes cheaper. But channel count sets aggregate bandwidth, and two of four channels populated leaves roughly half the bandwidth on the table regardless of the capacity sticker. Worse, nothing breaks. The runtime keeps working, because that's the job: offload layers, tolerate cache misses, fall back to generic kernels, page memory when it must. It finishes and reports a number, just a smaller one than the hardware should give. You won't see an error for this, only a benchmark that looks fine until you compare it against a properly populated board at the same quant and memory speed. Check the startup log for the channel count it actually detected, and confirm the kernel path matches the context length you're running, not the one from a demo.

Six numbers, not one

Before touching a launch flag, pin everything: model file, tokenizer, prompt template, runtime build, launch command, sampling settings. Run once cold, then warm enough times to expose caching, queueing, and thermal drift. Then split the stopwatch instead of trusting one total:

queue wait
model load
prefill
time to first token
decode rate
peak RAM / power / swap

Tokens per second is diagnostic evidence, not the deliverable: for a batch job, count valid completed records per hour; for chat, watch p50 and p95 first-token latency at realistic prompt lengths, since a config that wins a short warm run can lose the day once real prompts and model swaps show up. Change one variable per test and rerun the quality gate on every candidate, because quantization and context tricks can shave time off a run while quietly breaking the answer. Weigh the boring costs too: upgrade path, rollback, whether the box comes back identical after a disk failure, the kind of thing that makes a shared homelab LLM server miserable to hand off. Write the conclusion down with the workload, the date, and the condition that should trigger a retest: new model family, driver update, longer contexts, different traffic. Skip that and the number in your head turns into folklore.

Once the bandwidth math checks out, I stop short of the ceiling on purpose. I leave free memory for prompt variance, spare queue depth for the next interactive request, and thermal margin so a long batch doesn't throttle the box. That's capacity I could otherwise sell, and I give it up anyway, because a server pinned at its limit is one long prompt from a bad afternoon.