Quantize the KV cache before shrinking the model

Model weights are fixed the moment you load them. The KV cache keeps growing for as long as the conversation does, and on a card with eight or twelve gigabytes to spare, that difference decides which one runs out first. A 14B-class GGUF that sits fine in VRAM at an 8K context window can blow the same budget by 32K, not because the weights changed but because every extra token you generate adds a few more kilobytes of key and value tensors that have to stay resident.

Weights and cache are not the same knob

At short context, weights are the thing eating your VRAM. At long context, the cache is. Two consumers, one budget. Mixing those up is the single most common way people waste an afternoon: they drop the weight quant another notch, watch the model get dumber, and the 32K session still falls over anyway, because the thing that actually grew was never the weights. Dropping weight precision is a legitimate move too, just a different decision, the kind covered separately in picking a GGUF quant level.

Before touching either knob, write down what the session actually is. An interactive chat that runs for an hour behaves nothing like a code-completion loop firing every few seconds, and neither behaves like a document-extraction job chewing through a long PDF or an overnight batch nobody is watching. Each shape stresses the cache on its own schedule, and until you know which one you're running, "it worked" is not a target you can hit or miss.

One good run proves nothing

The actual test is a comparison: run the same retrieval, code-editing, and exact-copy tasks against a few cache dtypes with the weights held fixed, and log memory on every pass. Use identical inputs across runs and save the launch command next to the result, because six months from now you will not remember which flags produced which number. One warm sample tells you almost nothing. Include a cold start if your users will ever hit one, run long enough to expose thermal throttling or request queueing, and write down whether the output quality held, not just whether the process kept running.

What actually belongs in the log:

• time to first token
• prompt-processing speed and generation speed, tracked separately
• peak memory, and wall power too if you're set up to measure it
• task success against a quality gate, not a gut feeling about the output

Report medians, since that's the case you'll live in day to day, but keep a slow percentile sitting next to it. The p95 is what catches the pause that makes a demo look broken even when the average looks fine.

task:        retrieval / code-edit / exact-copy
cache dtype: fp16 baseline vs int8 vs q4
inputs:      fixed set, versioned, cold + warm
metrics:     ttft, prefill tok/s, decode tok/s, peak mem
quality:     pass / fail against the fp16 baseline
verdict:     keep, revert, or retest after the next upgrade

Shrinking the model is the lazy fix

The trap I see most often is solving a memory problem that only shows up in long sessions by cutting model quality everywhere, all the time. Local inference tolerates a lot of configurations that technically run, and "it loaded" is not a performance result any more than "the answer looked fine" is an evaluation. Check the runtime logs and the operating system's own memory counters, not the flag you happened to pass on the command line. Change one variable at a time and be ready to say why you expect the result to move, because otherwise what you're building is a pile of anecdotes with a chart glued on top.

There's a maintenance cost too, and it's the boring kind. A fragile few-percent win vanishes the next time the model, the driver, or the runtime gets updated, and nobody notices until the workload that used to fit stops fitting. A short script, a handful of representative prompts, and a plain-text results file are enough to retest after an upgrade. You do not need a dashboard for a single workstation.

Test the smallest plausible change first: touch the cache dtype, not the weights, and check it against the tasks that are sensitive to precision loss, retrieval and exact-copy, rather than the ones that shrug it off. Stop tuning once you hit your latency and quality target, and keep whatever spare capacity you didn't use. It absorbs a longer prompt, a background app you forgot was running, or whatever model you pull down next month. That's the same headroom instinct worth keeping on purpose, not just whatever happens to be left over by accident.

If I keep exactly one rule out of all of this, it's this: touch the component that is actually consuming the marginal memory, not the one that happens to be easiest to shrink. For a long-context session that component is the cache. Quantize that first.