Choose a local model by active parameters, not its logo
Pick the model whose active parameters and measured memory footprint fit the machine you have, not the parameter count printed in its name. That's the decision rule. The rest of this piece earns it: dense and mixture-of-experts models put very different pressure on memory, compute, and storage, and a name like "14B" doesn't tell you which kind you're about to feel. A dense 14B model does roughly 14B worth of work on every token. A larger MoE model with a bigger headline size might route each token through a much smaller slice of itself, so two models can wear the same number and describe different jobs for your GPU.
Total weights and active weights are different specs
Total weights are what has to live somewhere, on disk or in RAM, waiting its turn. Active weights are what the accelerator multiplies against your input on a given pass. Confuse the two and you size a card for a workload that was never going to run the way you assumed. Before touching a launch flag or buying a bigger card, write down what has to get better: first-token latency, accepted jobs per hour, how much else stays resident alongside the model, energy per completed task rather than per token, or fewer corrections a human makes afterward. "Make it faster" is not a target. It's a mood, and a mood won't tell you when you're done, or whether your change just cost you some quality.
Pin everything, then run the ugly inputs
Before comparing anything, pin the model artifact, tokenizer, prompt template, runtime build, launch command, and the sampling settings; any one drifting between runs invalidates the comparison. Use a small set of real workload inputs, including the awkward ones nobody wants to test, the long document, the malformed input, the conversation that's already run long. Run once cold, model unloaded and storage cold, if users will hit that path, then run warm repeatedly, long enough for cache effects, queueing, memory pressure, and thermal throttling to surface. You're measuring loaded memory, prompt speed, decode speed, and whether the output clears your quality bar, not the nominal size on a model card. Track it by phase: queue wait, load or activation, prefill, first token, decode rate, completion time, peak RAM, VRAM, power, swap, and whether the answer passed, needed a retry, or got repaired by hand. Raw tokens per second is a fine diagnostic and a poor final answer: count completed records per hour for automation, correction time for coding, p50/p95 first-token latency over long conversations for chat. A setup that wins a short warm benchmark can lose the day once real model swaps and long prompts arrive.
Your runtime will quietly rescue a bad config
The recurring mistake is assuming fewer active parameters means the whole model fits into less VRAM, when the runtime will happily keep going even when it doesn't: it offloads layers to system memory, pages things in and out, eats cache misses, queues requests, or falls back to a generic kernel instead of the fast path you thought you'd enabled, the same mechanism that lets a huge MoE get offloaded across a pile of host memory just to fit at all. The system keeps answering, just slower, and the slowness looks like the model's fault rather than the configuration's. Read the startup log. Check device placement. Watch the operating system's counters instead of the framework's summary, and confirm the optimization you asked for is active for the tensor shapes and context length you're actually using, not somebody else's. Change one variable at a time unless you're deliberately comparing two complete systems, and keep the actual outputs from every run, not just the timing numbers: quantization, context compression, sampling, and model swaps can all make an answer shorter or faster without making it correct. Run the quality gate on every candidate. If an answer got shorter, check it's still complete; if an extractor got faster, count records that are actually valid, not ones that merely parsed.
Write the number down before it becomes folklore
Weigh operational cost alongside performance: how the change affects startup, survives the next upgrade, stays observable, rolls back cleanly, or lets you rebuild the same server after a disk failure. A gain resting on an undocumented patch or manual cache-warming every shift is a bad trade for anything you don't babysit. Boring configurations age well; clever ones get inherited by whoever wasn't in the room when you made the trade.
Choose from measured resident memory and useful throughput on the runtime you'll run in production, not a spec sheet, and write the conclusion down: workload, date, reason, in the file the next person opens. Write the retest trigger beside it, because model families keep replacing each other faster than most runbooks keep up: a driver update, a longer context requirement, a new user, a different traffic mix. Skip that boundary and an old benchmark number quietly turns into infrastructure folklore nobody wants to question.
Once you've hit the target, leave headroom on purpose. Free memory absorbs prompt variance and whatever else shares the box. Spare queue capacity keeps a batch job from starving the interactive user in front of it. Thermal and power margin keeps the machine behaving the same way at two in the morning as it did during your test. That's the tradeoff I take every time: I give up the last slice of raw throughput, the number that looks best on a benchmark table, for a system that behaves the same way tomorrow as today. A fully packed accelerator makes a great demo and a bad neighbor.