Evaluate the adapter against the base model

Say you've got a base local model on disk and three or four LoRA checkpoints sitting next to it, each one trained on a slightly different slice of your own data, and somebody on the team wants a yes or no on which one goes into production. That's the actual situation this is about, not a leaderboard number and not a paper's headline claim. The honest unit of measurement isn't the model at all. It's a finished task running on a machine that also has other jobs to do: input goes in, an answer comes out, a quality gate passes it or it doesn't, and everything else on that box keeps responding while it happens. Optimizations that look great in isolation have a habit of vanishing the moment you put them back inside that loop.

What makes adapter evaluation genuinely harder than swapping a runtime flag is that a LoRA doesn't bolt on one clean new skill. It nudges the whole behavior distribution of the model, a little on everything, well past the handful of examples you actually trained it against. So before you touch hardware or flip a launch setting, write down in plain words what actually has to improve. First-token latency. Accepted jobs per hour. How many models you can keep resident at once. Energy per completed task. Fewer corrections from whoever reviews the output. Pick one, maybe two. "Make it faster" doesn't tell you when you're done, and it does nothing to protect quality while you chase it.

A baseline you could rebuild after losing the disk

Pin the model artifact, the tokenizer, the prompt template, the runtime build, the exact launch command, and every sampling setting. Not "roughly the same," pinned. Pull a small set of inputs from the real workload, and make sure the awkward ones are in there, not just the clean demo cases. If your users will ever hit a cold path, run it cold once: unloaded model, cold storage, the works. Then run warm, and run it long enough to expose cache effects, queue buildup, memory pressure, and thermal throttling, because none of those show up in the first thirty seconds.

Score everything side by side in the same pass: target tasks, neighboring tasks the adapter wasn't tuned for, safety boundaries, latency, and output stability. Record the phases separately instead of collapsing them into one total, because the total hides exactly where the time actually goes:

• queue wait before the request is even picked up
• model load or activation
• prompt processing and prefill
• time to first token
• decode rate and total completion time
• peak RAM, VRAM, power draw, and swap
• quality pass, retry, abstain, or repair

Raw token-per-second speed is useful diagnostic evidence and nothing more, it is not the product result. For an automation pipeline, count valid completed records per hour. For a coding assistant, fold in review and correction time, not just generation time. For chat, look at p50 and p95 first-token latency under conversation lengths that resemble what people actually type, not a one-line prompt. A setup that wins a short, warm, batch-of-one benchmark can still lose the actual day once model swaps and long prompts show up, which they will.

Local runtimes degrade quietly, they don't crash

The mistake that shows up over and over is testing only the examples that motivated the fine-tune in the first place. Local runtimes are, in a way, too well-behaved: they'll offload layers, page memory, eat a cache miss, queue a request, or silently fall back to a generic kernel rather than throw an error. That resilience is genuinely useful in production and it is also exactly what hides a degraded configuration from you. Read the startup logs. Check device placement. Watch the OS-level counters. Confirm the optimization you think is active is actually active for the tensor shapes and context lengths your workload really uses, not the ones from the demo.

Change one variable at a time unless you're deliberately comparing two complete, fully-configured systems against each other. Save the outputs, not just the metrics, and repeat every sample more than once. Quantization, context compression, sampling changes, and the model swap itself can all quietly change the content of an answer while making it look faster on a stopwatch. That's the whole reason a quality gate has to run on every single candidate configuration, no exceptions, and it's worth building that gate as a proper evals-and-llm-as-judge setup rather than eyeballing outputs. If a shorter answer shows up, check that it's actually complete. If an extractor got faster, count valid records, not braces that happened to parse.

The regression budget is the real decision, not the win

Operational cost sits right next to raw performance, not below it. Look at what the change does to startup time, upgrades, observability, rollback, and your ability to rebuild this exact server from scratch after a disk failure. A five percent win that depends on an undocumented patch, or on someone manually warming the cache every morning, is a bad trade for anything shared. Boring configurations age well. Clever ones need a maintainer who remembers why they're clever.

The rule that holds up is simple to state and annoying to actually follow: require a declared gain on the target cases, and set a tolerated regression budget for everything else, in advance, not after you've already seen the results you wanted. Write the conclusion into the result file along with the workload, the date, and the reason. Then write down the condition that should trigger a retest: a new model family, a driver update, longer contexts showing up in production, a new class of user, a different traffic mix. Skip that last part and last quarter's benchmark numbers quietly turn into infrastructure folklore that nobody can defend and nobody wants to be the one to re-run.

Leave slack in the system on purpose

Meet the target and then stop, with room left over. Free memory absorbs prompt variance you didn't test for. Spare queue capacity is what keeps a batch job from starving your interactive users. Thermal and power margin is what keeps the thing running the same way at hour six as it did at minute one. Efficiency isn't packing every resource to a hundred percent, it's finishing useful work predictably on the cheapest, least fragile system that actually clears the bar you set.

That's the real advantage of running this stuff locally instead of behind an API. Every layer is inspectable, so performance doesn't have to stay a vendor's claim you take on faith. Next, I'd pull the startup log for whatever configuration is about to ship and check that the optimization I think is active is genuinely active at the context lengths production actually sends, not the short prompts from the eval set.