Do not compare models with one universal prompt

Every model swap on a shared card costs you seconds before the first token even queues, and that has nothing to do with which model answers faster once it's resident in memory.

That's a hardware fact, not the problem I want to talk about. The problem shows up once you're comparing three local instruction models behind one API and sending them the same prompt because it feels fair. Identical visible text stops being identical input the moment role markers, system-message conventions, or recommended instruction formats differ. You end up scoring how gracefully each model degrades under a prompt it was never tuned to read, then calling that a comparison.

freeze everything except the template

Pin the model artifact, tokenizer, runtime build, launch command, and sampling settings. Let the one thing allowed to change per model be the prompt template it actually shipped with, treated like prompt-as-code: versioned and reviewed, not left to drift. Feed every candidate the same small set of inputs from your real workload, awkward cases included. Run the first pass cold, storage untouched, model unloaded, the way a real request meets an idle server. Then run warm, long enough for cache effects, queueing, memory pressure, and thermal throttling to show up.

Once runs are underway, stop trusting one wall-clock number and log per phase instead:

queue wait
model load / activation
prefill (prompt processing)
time to first token
decode rate, total completion time
peak RAM, VRAM, power draw, swap
pass / retry / abstain / repair

Raw tokens per second is a fine diagnostic. It is not the thing anyone downstream cares about:

• automation: valid completed records per hour, not tokens produced
• coding: review and correction time counts, or the number is fiction
• chat: p50 and p95 time-to-first-token across realistic conversation lengths

A setup that wins a short, warm, batch-of-one benchmark can still lose the day once model swaps and long prompts arrive together.

the runtime that never complains is lying to you

The mistake I see most: force every candidate through the prompt the old model liked, get an answer from each, and file that under compatibility. Local runtimes absorb damage without telling you: they offload layers to CPU, page memory around, eat a cache miss, queue behind another job, or drop to a generic kernel rather than error out. That keeps the demo working and can hide a configuration running at half of what it should. Read the startup log. Check where the tensors landed. Watch the OS-level counters, and confirm the optimization you enabled is active for the real context length and tensor shapes you're using.

Change one variable per run unless you're comparing two finished systems end to end, and keep every output, not just the metrics. Quantization, context compression, sampling tweaks, and model swaps can each make a system faster while quietly changing the answer, so the quality gate has to run against every candidate configuration, not just the baseline: evals-and-llm-as-judge, turned inward on your own infrastructure. If a shorter answer wins on latency, check it's still complete. If an extractor wins on speed, count records that validate, not braces that merely match.

Put operational cost next to the performance number, because they're the same decision: startup, upgrade path, live observability, rollback, and whether you could rebuild this exact server the day a disk dies. A five-percent win that depends on an undocumented patch or a human keeping the model warm is a bad trade beyond one machine you personally babysit. Boring configurations age well. That's the actual goal, not a consolation prize.

write down when this stops being true

Compare equivalent intent, not identical bytes on the wire, that's the rule that holds up. Write it into the result file with the workload, the date, and the reason you accepted it: a number missing those isn't a result, it's a rumor. Note what would invalidate it: a new model family, a driver update, longer contexts, a different user, a different hour's traffic. Skip that and last quarter's numbers quietly turn into infrastructure folklore that nobody can defend and nobody wants to re-run.

Leave slack after you hit the target, not before. Free memory absorbs prompt variance you never tested for. Spare queue capacity keeps an interactive user from feeling a batch job land on them. Thermal and power margin keeps this morning's number still true six hours later. Packing every resource to full is not efficiency, it's a system with no room left to be wrong.

The real advantage of running these models yourself is that every layer stays inspectable, so performance never has to be someone else's marketing claim. Measure the workload you actually have. Keep the quality gate honest. Stop tuning the moment the system is good enough to be boring.

One rule, and it's the only one worth keeping: compare what each model was asked to do, never the raw text you happened to send it.