Wake the GPU server only when work arrives

The workstation with the GPU sits dark most of the day. Next to it, a mini PC stays on around the clock, drawing next to nothing, waiting for a request. That request might be a private chat someone opens twice an hour, a coding loop hammering an endpoint every few seconds, or a nightly extraction job that runs while nobody's watching at all. Wake-on-LAN is what makes this workable: the small machine holds the network address, the client waits a beat, and the GPU box spins up only when there's an actual token to generate.

Hardware that pulls real power at idle doesn't have to run nonstop to feel available. Split the workloads by what they actually need and the bet holds up. A private chat doesn't care if the machine took a breath between messages. A coding loop mid-refactor, firing requests back to back, cares a great deal. A nightly batch job cares about something else: whether it finished by morning.

Timing the wake, not just the model

Before changing anything, write down what you have: one named model artifact, one fixed prompt set, the exact command that launches the server. Skip nothing, imply nothing. It feels tedious right up until a driver update or a quantization swap makes last month's numbers meaningless, and you're glad you wrote the launch flags down instead of trusting memory.

Then send the first request. Let it wake the host, poll health until the server actually answers, and give the client something honest to look at instead of a spinner that reads like a hang. Someone who sees "waking GPU host" with a rough estimate tolerates the wait. Someone who sees nothing hits retry, and now two boot sequences are fighting over the same GPU.

From there, follow the request all the way through instead of grabbing one number and calling it done. Time to first token catches loading plus prompt processing. Steady token rate describes decoding once the model is actually warm. Completion time is what the person or the job experiences, and it's usually the only number anyone outside the system cares about. Add peak memory, queue delay, and wall power when they change the decision, not as a default. For the nightly job, count completed valid jobs per hour. For the chat session, count the slow waits a person will remember, not the average that buries them.

I keep this as a table rather than a paragraph, because a paragraph lets me skip a column when I'm in a hurry:

Drop that last column and the table turns into trivia nobody rereads.

Where the boring config earns its keep

The failure I'd actually worry about isn't the wake taking too long. It's hiding a ninety-second boot behind a client that retries silently until the whole thing looks broken. That bug survives review because the system still hands back plausible text once it finally wakes up. Nothing crashes. Nothing logs an error. It just looks slow in a way nobody downstream can explain.

Local inference has a whole family of failures shaped like this one. A model partly offloads to CPU and throughput quietly halves. A cache misses and doesn't say so. Swap creeps up over a session. A queue keeps holding requests from clients that disconnected minutes ago. A fallback path changes where inference runs and moves your data across a boundary you meant to keep closed. None of these raise an exception. All of them show up in runtime logs and OS counters if you're watching while the test runs.

So watch the logs, not just the stopwatch. Change one variable at a time unless you're deliberately comparing two complete configurations against each other. Run it more than once, because a single fast pass tells you almost nothing about whether you got lucky. And read the actual output rather than trusting that speed implies the answer is still right: if a change makes an important task worse, that cost belongs in the same table as the gain, not in a footnote you write later and probably won't.

Operational simplicity deserves its own line item. A clever optimization that needs manual repair after every driver or model update isn't free, whatever the benchmark says. I'd rather run a boring setup I can rebuild from a service file or a short script than a fast one only I know how to fix at midnight. Keep the raw run artifacts out of the written report, but keep the hashes and the exact commands, so a rerun months from now still means the same thing.

The rule that's actually held up for me is to optimize availability and energy together across a month rather than treat uptime as a switch that's either on or off. It produces a stack with limits you can see, and a visible limit is something you can route around or schedule for, instead of just absorbing it. An invisible one turns into someone stuck waiting mid-afternoon, wondering why the assistant is slow, followed by an unplanned hardware order. Stop tuning once the workload clears its latency and quality target with real headroom left over. That headroom isn't wasted capacity sitting around for no reason; it's what absorbs the longer document, the extra concurrent user, or whatever the next runtime release changes underneath you.

Next time I touch this setup, I'm not benchmarking the model again. I'm putting a stopwatch on the wake sequence itself: how long from the first queued request to a health check that actually passes, and how much of that stretch the client spends showing the person nothing at all. Anyone still deciding how to build the workstation side of this in the first place should read the notes from setting one up before worrying about any of the above.