The layer-offload math nobody explains

In January I spent two weeks convinced that 32B models were a lie on 24 GB cards. My used RTX 3090 was producing 6.8 tokens per second from a 32B coder model at Q4_K_M while people with the identical card reported high teens. I drafted half a blog post about benchmark inflation. The model was fine. Two of its layers were running on my CPU.

This article is the napkin math I wish someone had shown me first.

what a spilled layer actually costs

A dense transformer generates each token by running it through every layer in sequence. A 32B model in the current Qwen class carries 64 transformer layers (65 offloadable blocks as llama.cpp counts them), and Ollama inherits that accounting. Generation speed is dominated by memory bandwidth: every token reads each layer's weights from wherever they live. My 3090 reads VRAM at a bit over 900 GB/s on paper. The dual-channel DDR4 next to it manages call it 45 to 50. Roughly a 20× gap per layer.

The part nobody spells out: those layer times add in series. Token 41 can't start until token 40 has finished the whole trip, so there's no parallel rescue. Your tokens per second is the reciprocal of the sum of every layer's read time. That sum does not care that 97% of your layers are fast.

the napkin

Just under 20 GB of Q4_K_M weights spread over 65 blocks is about 300 MB per layer. At 900 GB/s, a GPU layer costs roughly a third of a millisecond; 65 of them is about 22 ms per token, a ceiling somewhere in the forties of tokens per second before overheads. Now spill two layers to system RAM at 50 GB/s: those two cost about 6 ms each, and the per-token time goes from 22 ms to around 34. You just spent a third of your speed on 3% of the model.

Offload isn't a slope, it's a cliff, and the cliff starts at the first spilled layer.

Reality is worse than the napkin, not better. Activations hop across PCIe at every GPU-CPU boundary, the CPU-side threads fight your other processes for memory bandwidth, and long contexts add KV-cache reads on top of the weights. The napkin predicted a third; my measured drop was closer to two thirds: 6.8 against the 19.5 I eventually got back. Treat the arithmetic as the floor of the damage, never the ceiling.

why it spilled when the weights fit

The trap that got me: the weights alone did fit. Just under 20 GB into 24. But I'd set num_ctx to 16384, and the KV cache for a 32B at that context adds a few gigabytes, plus compute buffers, plus a sliver for the CUDA context itself. Ollama doesn't error on the shortfall. It quietly drops a couple of layers to CPU and keeps serving, slower. The only tells are ollama ps and the server log.

ollama ps                     # want: "100% GPU"
journalctl -u ollama | grep offload
# llm_load_tensors: offloaded 65/65 layers to GPU

The KV arithmetic deserves its own article, and it has one: local-kv-cache-math. Short version: context length is a memory allocation, not a preference, and it competes with your layers for the same 24 GB.

quant down before you spill

My fix ladder now, in strict order. First, cut context to what the task actually needs; halving 16k to 8k freed a couple of gigabytes on its own. Second, step the quant down: Q4_K_M to Q4_K_S bought me the last gigabyte, and picking between adjacent quants is a smaller quality risk at this level than people fear. Only third, and only for overnight batch jobs where I'm not watching a cursor blink, do I accept CPU layers at all.

With 8k context and the smaller quant, all 65 blocks landed on the GPU, and the same model on the same card did 19.5 tokens per second sustained. Somewhere between 2× and 3×, from moving two layers.

Nothing about the model changed. I owe it an apology.

There's a floor to the quant ladder, though. I won't drop below the Q4 class just to force a fit. In my own testing, the answers degrade in ways that cost me more time than slow tokens ever did. If it doesn't fit at Q4 with the context you need, run a 14B instead, or go shopping for different hardware with your eyes open.

where the napkin tears

Fair warning about the model I just sold you. Prefill is batched and parallel, so partial offload hurts generation far more than prompt reading, which is why a spilled setup reads your prompt briskly and then crawls, a pattern that helped me blame the wrong thing. MoE models change the arithmetic too, since only the active experts get read per token. And Apple's unified memory doesn't have this cliff at all (a different trade entirely). For dense models on a discrete GPU, though, the napkin has explained the mystery every single time I've seen it.

Check ollama ps before you blame the model. The model can't defend itself.