Picking a quant: the twenty minutes that decide everything

Open any GGUF repo on Hugging Face and the files tab greets you with fourteen or more variants, Q2_K up through Q8_0 plus an f16 nobody downloads. Whatever you click in the next twenty minutes fixes your speed, your context budget, and how smart the model actually is on your work. Model choice gets endless discussion; quant choice mostly gets a shrug and a habit.

The theory behind K-quants is genuinely interesting and almost useless for deciding. Here's the feel version instead, from a couple of years of running these things on a 24 GB 3090 and a 64 GB MacBook.

the menu, by feel

• Q4_K_M: the default. If you know nothing else about your situation, take it and move on.
• Q5_K_M and Q6_K: pay for these when the task punishes small mistakes, which in practice means code and structured output, and only when the VRAM is genuinely spare.
• Q8_0: I've stopped pretending I can tell it from Q6_K in real work. It mostly buys disk usage and load time.
• Q3 and below: the false economy shelf. More on that in a minute.

Modern repos also carry imatrix variants (quants calibrated on sample text so the same file size does less damage). When one exists at your chosen level, take it; it's the closest thing to a free lunch on this menu. One wrinkle from my notes, though: the IQ-series files ran noticeably slower on my MacBook's Metal backend than plain K-quants of the same size, while on the 3090 I couldn't measure a difference. Worth a two-minute check on your own hardware before a big download.

the blind test that humbled me

In early June I caught myself repeating quant folklore I'd never once verified, so I ran a blind test. Same 14B model at Q4_K_M and Q6_K, two Ollama aliases with neutral names, and a note about which was which sealed in my journal. For summarization work, meeting notes, RFC digests, I could not tell them apart. Six guesses over three days, three right. A coin.

On code I could tell inside an hour.

The Q4 alias invented a keyword argument on a library call and fumbled a regex character class that the Q6 alias, on the identical prompt, got right. Over the week I logged eleven coding tasks where the outputs diverged and correctly named the bigger quant nine times. Whatever the perplexity charts say, the damage lands first where syntax has no slack.

the false economy runs in both directions

Q3 is where I stopped going down. Squeezing a 32B into 24 GB at Q3 looks clever until you remember context needs room too, and now you have a dumber model and a cramped cache at the same time. For code I'd rather run a 14B at Q5 than a 32B at Q3, and I'll take that argument with anyone. For summaries the blind test says I genuinely can't tell, which argues for the smaller model anyway, since it's quicker to keep resident.

My embarrassing mistake ran the other way. Last winter I pulled a 32B coding model at Q8_0 (call it 35 GB) onto the 24 GB card because I wanted quality. A third of it lived in system RAM, generation crawled at single-digit tokens per second, and I spent three days telling myself the quality was worth it while quietly starting to avoid my own tooling. Dropping to Q4_K_M put the whole thing back on the GPU: output I honestly couldn't distinguish, several times the speed, and I stopped dreading the tool. Chasing quality had made my actual work worse.

A quant level is a context budget wearing a quality costume.

the worksheet I run in my head

The file size is approximately what the weights occupy in memory, so start there. Add the KV cache, which grows with context, for the mid-size models I run, another 1.5 to 3 GB per 8k of context unless the cache itself is quantized. Add roughly a gigabyte for runtime overhead and whatever else the card is holding. Then leave real headroom, because the failure mode when you guess wrong is layers spilling to CPU, and that tax dwarfs anything a smaller quant would have cost you.

Concretely, on 24 GB: a 32B at Q4_K_M lands around 19 GB, which fits, but with room for so little context that my so-called 32B daily driver was effectively an 8k-context model. The 14B at Q6_K plus 32k of context fits comfortably. That was the actual choice all along, and it took me an embarrassing while to see it: the quant picker is secretly a context picker.

My rules, stated plainly so you can disagree: Q4_K_M unless there's a reason, Q5 or better for anything that writes code I'll run (the local coding model roundup assumes that floor), and nothing below Q4 for billable work. And since every number above came off my two machines, measure on yours before you trust any of it.