Server CPU or desktop CPU for local models?
Somebody always buys the wrong box first. A retired dual-socket EPYC server looks unbeatable on paper, more memory channels than any desktop, capacity a model would never find in consumer RAM. Weeks later its owner finds an idle-power draw that spins the meter and a token rate barely past the desktop it replaced, because the workload was never bandwidth-bound. That's the failure this piece heads off: paying for channels and capacity when the job needed clocks, low idle power, a simple platform.
Stop treating the model as the thing you're optimizing. Treat the completed task as the unit of measurement instead, and the confusion clears on its own: evidence in, an answer out, a quality gate passed, the rest of the machine still responsive.
Name the bottleneck before you shop
Everything downstream depends on one fact you can check before spending a cent: is the model bandwidth-bound on system memory, partially offloaded between CPU and GPU, or is the CPU just coordinating GPUs doing the real math. Three different purchases, one shopping cart. Pick one.
A buyer weighing a used EPYC platform against a modern desktop is really asking which of those three situations they're in, and the honest answer takes an afternoon of testing, not a spec sheet. Before touching hardware, write down what has to improve: first-token latency, jobs accepted per hour, resident model capacity, energy per task, fewer human corrections. "Make it faster" doesn't survive a real workload.
If the bottleneck is partial offload, placement matters more than the chip badge. The arithmetic behind GPU offload is worth working through before you assume more cores fix a problem that's about which layers sit where.
Numbers only count if you can rerun them
A baseline only matters if someone else, future you included, can reproduce it: pin the model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Test a small set of real workload inputs, including the awkward ones. Run once cold, storage unwarmed, if users hit that path; then warm, long enough for cache effects and thermal throttling to show up.
Price the whole board against the whole result: chassis, memory, power, cooling, set against throughput on your workload, not a vendor's. Record phases separately, because one "total time" number hides where a box wins or loses:
| Phase | What it exposes |
|---|---|
| Queue wait | contention before work starts |
| Model load / activation | cold-path penalty, disk and RAM speed |
| Prefill | compute-bound stretch, favors clock speed |
| Time to first token | the number a user feels |
| Decode rate | bandwidth-bound stretch, favors channel count |
| Peak RAM, VRAM, power, swap | headroom and thermal risk |
| Quality pass, retry, abstain | whether the fast answer was right |
Tokens per second is diagnostic, not the product. Automation cares about valid completed records per hour, coding needs review and correction time folded in, chat needs p50 and p95 first-token latency across real conversations. A configuration that wins a short, warm, one-request benchmark can lose the day once model swaps and long contexts arrive.
The core-count trap and the failures that hide inside it
The recurring mistake is comparing core counts while ignoring memory channels and idle draw, as if a marketing number were the whole platform. Local runtimes are stubbornly willing to keep going: they offload layers, page memory to disk, eat cache misses, queue requests, or fall back to a generic kernel rather than fail outright. That resilience hides a degraded configuration, so check the startup logs, tensor placement, and OS counters, and confirm the optimization is active for your actual shapes and context length.
Change one variable at a time unless you're comparing two complete systems, and keep outputs, not just metrics: quantization, context compression, and sampling tweaks can make a run faster while quietly changing the answer. Run the quality gate on every candidate. A shorter response needs a completeness check; a faster extractor needs its records counted, not its braces.
Buy the platform for the bottleneck, then leave room to breathe
Operational cost sits beside raw performance: startup, upgrades, observability, rollback, and whether you could rebuild the server after a disk failure. A five-percent gain that depends on an undocumented patch or hand-warming the cache every morning is a bad trade for anything shared. A boring configuration is a good sign, not an insult.
The durable rule: buy the platform around the actual bottleneck, never the processor badge, whether that's a used EPYC chassis turned into a home LLM server or a desktop under a monitor.
- More channels and capacity: EPYC wins when the model is bandwidth-bound and needs resident capacity
- Higher clocks and lower idle power: the desktop wins when the workload is compute-bound or intermittent
- A simpler platform: the desktop wins again when nobody wants to own BIOS quirks and NUMA topology for one model
Write the conclusion into a result file with the workload, date, and reasoning, and write down what should trigger a retest: a new model family, a driver update, longer contexts, a different traffic mix. Skip that and old benchmark numbers calcify into folklore nobody questions.
Leave headroom once you've hit the target: free memory for prompt variance, spare queue capacity so a batch job doesn't starve an interactive user, thermal margin so the box runs a year instead of a month. A fully packed system was never the goal; one that clears the bar on cheap, sturdy hardware beats one pegged at full utilization and a bad week from falling over.
Every layer here is something you can inspect, which is the real point of running this locally: performance doesn't rest on somebody else's marketing slide. Measure your workload, keep the quality gate honest, stop tuning once the system is boring enough to trust.
I'd trust every number here only until someone changes the model family — and if you're the only user and this is a weekend project, skip the harness and buy the desktop; the rigor above is for machines other people depend on, not a box under your own desk.