Dense or MoE for local inference?

A 24 GB card will happily hold a dense 13B-class model, and it will choke on a MoE model with a similar active-parameter count, because the experts that aren't firing on a given token still have to live somewhere. That's the whole tension. Most people find it out after they've already downloaded the GGUF.

Weights on the wrong side of the bus

Mixture-of-experts routes each token through a handful of experts instead of the whole network, so compute per token drops. Memory does not drop the same way, because you don't know in advance which experts a token wants. On a 24 GB card with partial offload into system RAM, a chunk of the model ends up living behind a PCIe link that moves data far slower than the GPU chews through activations. The router doesn't care that half its picks are one hop from the CPU; it just asks, and something answers. That's not a driver bug. It's the shape of the hardware you bought, and it's most of the case for buying VRAM before more compute, a point worth chasing in hardware for local llms.

Timing the boring parts

Before touching a launch flag, pin the model artifact, tokenizer, prompt template, runtime build, and sampling settings. Run once cold, the way a real request hits it after the process has sat idle, then run warm enough to see caching and thermal drift. Don't trust one number for the whole run:

• queue wait before the request even starts
• prefill time on prompt lengths you see in production
• time to first token, kept separate from total time
• decode rate across the full completion
• peak VRAM, RAM, swap, and power draw
• whether the output passes your quality gate on the first try

Raw tokens per second is a diagnostic, not a verdict. What you want is completed, correct work per hour:

valid_completions_per_hour = completions_passing_gate / wall_clock_hours

A dense model that's slower per token but rarely needs a retry can beat a MoE setup that's faster and wrong twice as often. For chat, swap in p50 and p95 time-to-first-token; for coding, fold in your own review time, since a fast wrong diff still costs an afternoon.

Write the number down and move on

The easy failure is celebrating a fast active-compute figure while the idle experts stream across that slow boundary on every request. Local runtimes keep going anyway, offloading layers, missing cache, falling back to a generic kernel, rarely loud enough to notice without reading the startup log. Change one variable at a time, keep full outputs rather than just metrics, and rerun the quality gate on every candidate, since quantization and context tricks (see inference optimization) can shorten an answer without making it wrong. Write the decision down: which architecture fits the memory path for your workload, and what should trigger a retest, a new model family, a driver update, a longer context. Leave headroom above whatever number you land on.

None of this measuring is free. If your workload is small and stable, just try both, eyeball which one feels fine, and get on with your day. The elaborate baseline earns its keep once a wrong guess starts costing you something, and for plenty of hobby setups that day never comes.