The KV cache is eating your VRAM

the 01:40 crash

The ticket-triage agent had been running for just short of three hours when it died. Friday night in late May, on the homelab box in the hallway closet (an old 3090 with 24 GB), grinding through a client backlog with a 32B Qwen coder at Q4_K_M. The weights are just under 20 GB, resident since ten in the evening. At 01:40 the llama-server log ended with a CUDA out-of-memory error, and my wrapper script helpfully restarted it into the same wall forty-one times before I woke up.

I blamed the quant, spent Saturday evening pulling a smaller one, and re-ran. It died too, about an hour later than the first. Only then did I read the actual failed allocation in the log: a single 8 GB block. Weights don't allocate like that mid-run. A KV cache does.

The weights were innocent.

My harness had a fallback I'd written in March and forgotten: when a session overflows its context, it doubles num_ctx and reloads. 8k held for the first two hours, one gnarly ticket thread pushed it to 16k, and the next doubling asked for 32k. llama.cpp allocates the entire cache up front at whatever size you request, so the restart tried to put an 8 GB cache next to 20 GB of weights on a 24 GB card. You can do the subtraction.

what the cache actually stores

Every token in your context stores two small vectors per layer, a key and a value, so attention can look back without recomputing the past. Per token that's 2 × layers × KV heads × head dim numbers, two bytes each at fp16. The 32B I run has 64 layers, 8 KV heads, head dim 128: roughly a quarter megabyte per token. Sounds harmless until you multiply. 8k tokens is 2 GB. 32k is 8 GB. The advertised 128k would be 32 GB: more than the entire card, before a single weight loads.

Grouped-query attention is the only reason the numbers are even that kind. Eight KV heads instead of the full forty; older models without GQA are several times worse per token. Dropping to an 8B doesn't rescue you either: about 128 KB per token, so a full 128k context still costs 16 GB of cache for a model whose weights fit in under 5 GB. The cache outweighs the model three to one.

Advertised context is an architecture spec. Your context is whatever VRAM is left once the weights move in.

the fantasy of 128k on 24 GB

Run the same arithmetic for a 70B-class model and the cache alone lands north of 40 GB at full context. That's academic, since the weights never fit a 24 GB card in the first place, but people keep asking anyway. The honest version for my 32B: after weights and a compute buffer, about 3 GB remain, roughly 12k tokens at fp16. The model card says 128k. I wrote about the paid-API flavor of this gap in context windows are a lie; locally the lie is more literal, because the allocator eventually calls it.

Everyone budgets for weights. Almost nobody budgets for the part that grows with the conversation.

quantize the cache, not just the weights

The fix that saved the triage agent is KV cache quantization. llama.cpp can hold the cache at q8_0 instead of fp16 (half the bytes) or q4_0 for a quarter. Under Ollama it's two environment variables:

OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q8_0

Flash attention is required for the quantized value half, and that's the catch: some architecture and backend combos quietly fall back to fp16, so verify the cache type in the startup log rather than trusting your env vars. On q8 I couldn't tell two weeks of triage diffs apart from the fp16 baseline: same fixes, same style. q4 was a different story: long-range recall got flaky, and asked about an identifier from thirty-odd thousand tokens back, the q4 run misnamed it in two of my nine test threads. q8 is my default now. q4 I treat as an emergency, not a setting.

the arithmetic I do before every run

Card memory, minus weights, minus about a gigabyte for the compute buffer, is your cache budget. Divide by the per-token cost (quarter MB at fp16 for a 32B, half at q8) and that's your true num_ctx. Set it explicitly and make the harness respect it. Parallel slots each take a slice of the total, so two slots halve your per-session context again. It's the same budgeting discipline as the GPU offload math, pointed at the half of VRAM nobody draws on the diagram.

The 3090 now runs triage at 24k with a q8 cache and a forced summarize-and-restart at 20k, a trick lifted from the context budget playbook on the paid side. Zero OOMs in five weeks. The backlog doesn't care either way, but I sleep through the night again, which was the actual feature request.