Restart a local model server without dropping work
The rule is short: stop admitting new requests before you touch the process, let whatever is in flight finish inside a hard time cap, bring the replacement up, confirm it is actually ready, then swing traffic over. Skip any one of those steps and you pay for it, not in a benchmark, in a support channel. Everything below is just the case for why the order matters, because on paper it looks like overkill for a single kill command.
Admission stops before anything else does
Picture one machine running a shared model endpoint behind a lightweight reverse proxy. On it: a private chat opened twice an hour, a coding loop firing in bursts, and a nightly batch job extracting from a queue of documents. Three tolerance profiles on one process, ordinary outside a lab. Most write-ups open with a model name or a peak throughput number. Wrong starting point. What actually decides how you should restart is the work waiting on the other side of the API at that moment. Even a one-machine service can avoid handing back failed generations during an upgrade. That's the whole constraint: no cluster or orchestrator needed, just stop accepting new work before tearing anything down.
What the stopwatch has to catch
Before touching the process, get a baseline while the system is still boring: one named model, one fixed prompt set, the exact server command and flags, plus the model artifact and prompt template. Those last two are what people forget, and losing either makes a before-and-after comparison meaningless weeks later. The restart itself follows the same order every time: announce draining, cap how long an in-flight request may run, keep the queue instead of dropping it, start the new process, and only flip traffic once it passes a real readiness check, not just an open port.
Measure the whole request, not just the model. Time to first token covers loading and prompt handling, steady token rate covers decoding, and completion time is what a person or downstream job actually feels, the only number most people should say out loud. Add peak memory, queue delay, and wall power when they would change the decision, not because a dashboard has room for them. Batch work gets judged on completed valid jobs per hour; interactive work on the slow waits a person remembers.
A benchmark nobody attached a decision to is trivia. A setting nobody owns is folklore.
Keep that discipline in one running log: artifact and flags, workload and inputs, cold and warm start at the percentiles that matter, memory and energy, quality failures or abstentions, and a decision with an owner and a retest date. Drop that last field and the log becomes numbers nobody trusts.
Failures that never throw
The move I would warn against is killing the runtime mid-stream and trusting every client to retry safely. It is tempting because it mostly works: the system keeps producing plausible text, so nothing looks broken. Local inference fails soft more often than loud. A model quietly offloads part of itself to CPU and slows down, no error thrown. A cache misses and nobody notices, because the answer still comes back, just later. Swap grows in the background. A queue keeps holding requests whose client already gave up. A fallback path kicks in and changes where the data goes, which matters more than a slow response once it isn't local. None of it has to raise an exception to hurt you.
Catching this means watching runtime logs and OS counters while a test runs, not just what your framework reports. Change one variable at a time unless you are deliberately comparing two full configurations. Run it more than once: a single good number is a sample of one, and you cannot tell a durable improvement from a lucky run without repeats. Read the actual outputs; speed alone isn't proof two configurations behave the same, and if a change makes some task worse, that cost belongs next to the gain.
Operational simplicity belongs in the same record. An optimization needing manual repair after every driver or model update was not free, however fast it tested. I'd rather run something boring and reproducible, from a service file, container definition, or short script, than something clever only one person can fix. Keep raw run artifacts out of the report, but keep the hashes and commands so a rerun months later still means the same thing.
Headroom you can actually spend
The rule I keep coming back to: treat a restart as a small deployment, not a command you fire and walk away from. Less fun than arguing over which model or accelerator to buy next, but it leaves a stack whose limits you can see, which is really just resilient service design applied to a box under a desk. A visible limit gets scheduled, priced, or handed to a queue. An invisible one turns into a person waiting too long, then somebody patching production at short notice.
Stop tuning once the workload clears its quality and latency target with real headroom left, and mean it. That margin gets used: it absorbs the next long document, the next extra user, a warm afternoon when the fans cannot keep up, or a runtime update installed without a fire drill. Efficient local LLM use is mostly spending capacity only where it changes what the user gets back.
None of this is worth doing if you're the only one who ever hits the endpoint. A solo box restarting mid-request costs one annoyed retry, nothing more, and a four-step drain ritual for that is a hobby dressed up as engineering. Save the ceremony for the machine other people depend on.