Have a retirement plan for local models

When you swap a local model for a supposedly better one, keep the old one loaded and routable until every downstream caller has moved off it. That's the whole rule. The rest of this is the argument for why teams skip it anyway.

Local replacement is a small migration, not a file rename: a service with saved prompt presets and a few downstream consumers is a graph of things quietly assuming the old model's quirks. Before touching hardware or a launch flag, name what has to improve: first-token latency, accepted jobs per hour, resident model count, energy per task, fewer reviewer corrections. "Make it faster" won't tell you when you're done, or protect you from a model that's quietly wrong more often.

Name the number before you touch the launch flag

Start with a baseline you can reproduce: pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings, then run a small set of real inputs, including the edge cases nobody likes testing. Run once cold, storage and model both unloaded, then warm long enough for cache effects, queueing, and heat to show up.

Inventory every caller, run old and new side by side on live traffic, version the endpoints, and set a removal date. Time each phase, not just the total: queue wait, model load, prompt processing, time to first token, decode rate, peak RAM, VRAM, power, swap, and the quality pass, retry, or repair.

Raw token throughput is a diagnostic, not the product. For automation, count valid completed records per hour. For coding, fold in review and correction time. For chat, watch p50 and p95 first-token latency across realistically long conversations. A setup that wins a short, warm, single-request benchmark can lose the day once swaps and long prompts arrive for real.

The failure that never sets off an alarm

The recurring mistake is overwriting a model alias in place and only later noticing the output schema, or the tokenization, changed underneath it. Local runtimes, Ollama included, are almost too good at staying up: they offload layers, page memory, miss a cache, queue, or fall back to a generic kernel rather than fail loudly. That resilience hides a degraded config, so check startup logs, device placement, and OS counters for whether the optimization is active on the actual shapes and context length.

Change one variable at a time unless you're comparing two complete systems, and save the outputs, not just the metrics. Quantization, context compression, sampling, and model swaps can change the answer while making it faster, so run the quality gate on every candidate: check a shorter response for completeness, count an extractor's valid records, not braces that happen to parse.

Operational cost sits beside performance: startup, upgrades, observability, rollback, reproducing the server after a disk failure. A five percent gain riding on an undocumented patch or manual cache-warming is a bad trade for a shared service. Boring configurations age well.

Put the decision next to the evidence, not in someone's head: workload, date, and reason go in the result file, along with what should trigger a retest, a new model family, a driver update, longer contexts, another user. Skip that and last quarter's benchmark becomes folklore nobody questions.

Leave headroom once you hit the target, and resist claiming the rest for a marginally better number. Free memory absorbs prompt variance, spare queue capacity keeps a batch job from starving an interactive user, thermal margin lets it run for hours, not minutes. None of that shows in a benchmark, which is why it's cut first. Efficiency isn't packing everything to full, it's finishing the work predictably on the cheapest system that clears the bar: squeeze the local tier, don't max it out.

I'd still say this: for plenty of setups, all of the above is more process than the stakes deserve. If a bad swap would sit unnoticed for a week and nobody would file a ticket, skip the parallel endpoints and the retest trigger. A text file naming which model is running, and why, kept honest, does the job.