Do the VRAM budget before downloading the model
Twelve gigs of VRAM and a 14B-class GGUF is the rig I keep coming back to for this arithmetic, not because it's special, because it's ordinary enough that the numbers matter.
The file size on a model card tells you almost nothing. It's the opening line of a bill that also covers the KV cache, runtime buffers, and a margin left empty. Skip one and the download still finishes, then fails later, quietly.
Four numbers before you hit download
Before touching a setting, name the actual job: interactive chat, code completion, document extraction, an overnight batch. Decide what a good result looks like, then work out four numbers, in order:
- the raw weights at the quant you're considering
- the KV cache sized to the context you'll use, not the max advertised
- runtime overhead: buffers and allocator slack
- a safety margin of about ten percent, because something always wants it
On a 12 GB card that math gets tight fast for a 14B model at real context lengths. Useful example, not a flattering one.
Where a GGUF's file size stops telling the truth
The common trap: picking a quant off its file size, then finding a long prompt pushes layers back into system RAM. Loading proves nothing about speed, and a plausible answer isn't the same as a verified one.
Watch the runtime logs and the OS memory counters, not the offload flag on the command line: the flag describes intent, the logs describe what happened. Change one variable at a time, or you're just collecting anecdotes; the offload math is worth doing by hand once.
Any win found this way has a shelf life: a five-percent gain from one driver build evaporates next update. Keep whatever you used to measure: a short script, a few prompts, a plain-text file of results. A single workstation doesn't need a dashboard.
What actually earns you headroom
Run every test with identical inputs and save the launch command with the result. One warm run tells you nothing. Include a cold start if a human will hit one, and repeat long enough for heat or queueing to show.
Record only numbers that change your decision: time to first token, prompt speed, generation speed, peak memory, wall power if it matters, task success. Medians describe the normal case; the slow tail is where the pauses live, and averages hide it.
None of this is about chasing the largest parameter count the catalog offers. Fit the model to the job, test the smallest change first, stop tuning once you clear your latency and quality bar. Leftover headroom absorbs the next long prompt and next month's model.
None of this replaces just buying a bigger card. A worksheet gets you a setup you understand on hardware you already own. It won't stop you wanting the next one anyway.