Read GGUF quant names without memorizing folklore
You've got a model page open with a dozen files between Q2 and Q8, all the same model, and the only difference is a string like Q4_K_M meaning nothing without forum folklore. That string is a recipe identifier, not a quality grade. Local AI talk starts from a model name or a throughput number; the real starting point is the work behind the API: a private chat, a coding loop, and a nightly job pulling something different from the same box.
Pick a baseline before you pick a winner
Before touching a config file, capture a baseline: one named model, one prompt set, the exact launch command, or you can't reconstruct the test. Then work from memory budget, skim the quantizer's notes on what drops first, and test two adjacent candidates on real prompts, not a generic set (gguf-quant-picking has more on that). Track it end to end: first-token time, decode rate, completion time, peak memory, queue delay, wall power, jobs per hour, or the waits people remember.
One table, and I never skip the last row.
| Field | Forces you to record |
|---|---|
| Artifact, runtime, flags | Exactly what you ran |
| Workload, fixed inputs | The prompts the numbers mean |
| Cold/warm start, p50, p95 | Not just an average |
| Peak memory, wall energy | What the box paid |
| Quality failures | Where it quietly gave up |
| Decision, owner, retest date | Who's accountable when it drifts |
A benchmark with no decision is trivia. A setting with no owner is folklore.
The failure mode doesn't throw an error
The tempting shortcut: treat every Q4 variant, across models and quantizers, as one tier, since it still reads fine. Local inference fails softly: partial CPU offload, a missed cache, swap growth, a queue holding dead requests, a fallback quietly shifting the privacy boundary. None of it raises an exception. Watch logs and OS counters while it runs. Change one variable at a time, except when comparing whole configs, repeat enough to rule out luck. Read the outputs. Speed alone proves nothing. A change that helps latency but wrecks a task gets logged, cost included. Manual repair after every driver bump isn't free. Boring and reproducible beats clever. Keep the hashes and commands, not the raw artifacts.
Where the margin goes, and where I'm still stuck
Record the exact artifact. Judge output, speed, and fit together, as one decision, not three. Less exciting than chasing the biggest model, but the limits stay visible: routed around, scheduled for. Invisible ones turn into unexplained waiting and a panic upgrade. Stop once it clears its quality and latency bar, with headroom for a longer document or the next runtime release. What I haven't worked out: after a driver or runtime update shifts your numbers, whether the quant is at fault or the engine is. The retest date admits it needs to happen; it doesn't say which knob moved.