Rank local models by quality per occupied gigabyte

Here's the rule, stated before any of the reasoning behind it: score the combination of models that stays resident together, not the single model that wins a leaderboard, because on a shared card the scarce thing is gigabytes, not accuracy points, and a 24 GB card asked to do coding, extraction, and chat at once is running a portfolio, not a bake-off. Everything below is the argument for that rule, plus the places it stops applying.

The reload tax

A 24 GB card carrying coding, extraction, and chat is a fine reference point precisely because it's ordinary: no lab bench, no claim that this one machine speaks for every setup you'll ever tune. What it does force is honesty about the job. Write the workload down first, interactive chat, code completion, document extraction, whatever it actually is, and decide what counts as a good result before you touch a single setting. Skip that step and you'll optimize whichever number happens to be on screen, usually the one from the model with the biggest benchmark score, while the thing actually constraining you goes untouched. The single best model on a quality chart can still be the wrong system choice, because loading it for every request means evicting whatever else was resident, and eviction is not free. Read up on what fits on a card sized for real workloads and the arithmetic gets clearer fast: planning capacity for one giant model and planning capacity for three specialists sharing a budget are different exercises entirely.

What earns a line in the log

The test that actually answers the question scores task suites, counts runtime overhead as part of the cost, and compares combinations that can sit in memory together rather than models in isolation. Same inputs every run. Save the launch command next to the result, because you will not remember later which flags produced which number. One warm sample tells you almost nothing; include a cold start whenever a human will hit one in practice, and run it long enough to expose heat throttling or queueing rather than stopping at the first clean number. Only a short list of figures can actually move the decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power where that matters, and whether the task passed or failed. Medians describe the ordinary case; a slow percentile is what catches the pause that makes someone stop trusting the tool. Notes on noise, responsiveness, and setup friction belong right next to those numbers, because friction is what determines whether a setup survives contact with a real workday.

Loaded is not the same as fast (models)

The trap almost everyone falls into is filling all the VRAM with one general-purpose model and quietly paying a reload every time a specialized request comes in. Local inference is full of configurations that technically run. That a model loaded is not a performance result, and that an answer looked plausible is not an evaluation, and conflating either one with a measurement is how teams end up defending setups nobody actually enjoys using. Check the runtime logs and the operating-system memory and power counters instead of trusting whatever flag you thought you passed. When two runs disagree, change exactly one variable and be ready to say why you expect the number to move; otherwise what you're building is a pile of anecdotes with timestamps on them, not a benchmark.

A win that has to survive Tuesday's driver update

None of this holds forever. A fragile five-percent win evaporates the next time a model, driver, or runtime changes underneath you, which happens more often than anyone plans for; keeping a record of what actually shifted between releases is cheaper than rediscovering the regression by hand after the fact. So the version of this discipline worth keeping is small: a short script, a handful of representative prompts, a plain-text result file you diff after every upgrade. That's the whole apparatus, not an observability platform bolted onto a workstation. The decision rule underneath it stays the one from the top: optimize the resident set for total workload value, test the smallest plausible change first, and stop tuning the moment the workflow clears its latency and quality bar. Leave headroom. Spare capacity is what absorbs the longer prompt, the background process you forgot was running, and the model you'll want to try next month.

None of which means the measuring itself is free. Every hour spent logging cold starts and diffing plain-text files is an hour not spent shipping, and if you're only ever running one model at a time with nothing else competing for the card, most of this ritual is overhead you don't need yet.