Run the model server under systemd

A machine that answers requests for other machines does not get to go dark just because you closed the laptop lid it was ssh'd in from.

That's the real argument for systemd, not init-system taste. What matters is the job on the other end of the API: a private chat checked twice an hour lives or dies on felt latency, a coding loop firing every few seconds needs completion time end to end, and a nightly batch job wants jobs finished per hour, all three on one box.

the tmux pane is not a deployment

Run the server in a terminal, close the terminal by accident, and the service is gone until someone notices. Reboot for a driver update and it stays gone too. Fine for trying a model out. Useless as the thing three other machines depend on.

Before wiring any of that into a unit, pin the comparison down: one named model, one fixed prompt set, the exact command, model artifact and prompt template noted alongside it. Skip that and next month's you calls a different setup a regression.

A few things then stop being optional: a dedicated user instead of your login, an environment file instead of command-line flags, startup ordering against the driver and the network, and a restart cap so a crash loop doesn't hammer the card.

Worth watching once it's up, kept short:

• time to first token, cold vs warm start
• steady decode rate, p50 and p95
• peak memory, wall draw if power matters
• queue depth, plus quality failures or abstentions

the failures that skip the error log

An API key or model path on the ExecStart line sits in shell history and process listings for anyone with a login to read. An environment file with tight permissions fixes that.

Most of what breaks here throws no error at all: a model partly offloading to CPU when it doesn't fit in VRAM, a sizing problem covered in hardware for local llms, a cache that quietly stops hitting, swap creeping up, a queue holding a disconnected client's request, and a fallback that quietly moves your data onto someone else's API the moment the local one hiccups. Catch it in the journal and the resource counters, not in a 200 response.

Change one variable at a time unless you're comparing whole configurations, and run it more than once before trusting a win. If the faster setting makes an answer worse, log that cost next to the gain: a benchmark with no decision attached is trivia, a setting with no owner is folklore.

one unit file, nothing else

A hand-patched script needing surgery after every driver update isn't free, it's borrowed time. The boring version, all in the unit file, is the one you can hand off. Keep the weights and logs elsewhere, but keep the hashes and commands so a re-run means the same thing.

A launch path that lives only in someone's memory is an invisible limit, and invisible limits turn into a 2am page instead of a line on a capacity plan.

Stop tuning once the workload clears its quality and latency bar with real headroom underneath, room for a longer document, one more user, a hot afternoon, the next runtime update.

If it didn't start from the unit file, it isn't running.