A/B test quants with your prompts, not a leaderboard

VRAM does not negotiate. Whatever quantization you load claims a fixed slice of it, and the KV cache, the context window, and everything else your workload needs has to fit in what's left. That's the real constraint, not a leaderboard entry. A Q8 that scores well elsewhere is useless without room for your context length. It's whether Q4, Q5, or Q8 earns its memory on your machine, on your prompts, today.

I keep one base model in three GGUF builds for this reason. Nothing exotic, no lab bench. Before touching a setting, write down the actual job, chat, code completion, document extraction, overnight batch, and decide in advance what passing looks like. Skip that step and you'll rationalize whatever number the quant produces; model-family averages smooth over the errors you'd notice day to day.

Strip the file names before you time anything

Freeze your sampling settings, then shuffle the outputs so you're scoring answers, not guessing which file made them. Same inputs every run, launch command saved with each result: "I think that was the Q5" is not a record. One warm sample tells you almost nothing. Run a cold start too, and repeat enough to expose thermal throttling. A quant that changes the answer along with the speed has become a different system, roughly the discipline behind evals-and-llm-as-judge run by hand.

Only record numbers that could actually flip the decision:

• time to first token
• prompt-processing and generation speed
• peak memory, wall power if you're metering it
• task success, scored the same way each run

Medians describe the typical run. A slow percentile catches the pause that makes the tool annoying, which the median hides. A plain-text log covers it:

job:        one named, repeatable task
inputs:     fixed, versioned
pass/fail:  scored the same way every run
latency:    cold, warm, p95
verdict:    keep, revert, retest

Loading successfully is not a result

The common trap is grabbing the biggest quant that fits, on the theory that more bits must be better. Local inference is full of setups that merely run. "The output looked fine" is a vibe, not an evaluation. Check runtime logs and OS-level memory numbers instead of trusting whatever flag you asked for. When two runs disagree, change one variable and say why, or the exercise turns into anecdotes with a spreadsheet attached.

There's upkeep too. A fragile five-percent win evaporates the moment a driver or model file changes. A small script, a handful of representative prompts, and a plain-text result file is enough to retest after every upgrade. Skip the observability stack; save that for problems the size of agent-observability, not a laptop running one model.

Keep the smallest quant you can trust

My rule: keep the smallest quant whose failures don't change the outcome. Test the smallest plausible step down first, leave headroom, and stop tuning once the workflow clears its latency and quality bar. That spare memory absorbs a longer prompt, a background process you forgot about, or next month's model.

Local models reward this restraint more than raw horsepower. Clear boundaries and a controlled comparison produce the first real gain, before you spend a cent on hardware. None of which guarantees my shelf is the right shelf for yours, or that this rule won't talk you into something a stricter eval would catch.