Use a mini PC as the control plane, not the muscle

Route everything through the small machine except the actual generation step, and keep the GPU box for exactly one job: producing tokens. That's the whole rule. Everything below is me earning it.

I run a low-power mini PC next to a GPU workstation that wakes on demand, the kind of box I've written about assembling as a homelab LLM server. I had it backwards for a while, workstation as the brain, mini PC as an afterthought. The mini PC is on all day; the workstation sleeps until something needs a big model, and whichever one runs coordination decides how the system behaves on an ordinary Tuesday, not a benchmark run.

Not every request deserves the big box

Local AI write-ups love a model name and a peak throughput figure, as if one number described every workload. It can't. A private chat opened twice an hour, a coding loop firing off short completions while you edit, and a nightly extraction job chewing through a folder of documents are three workloads with three different tolerances for latency, and treating them as one job is how a good pitch turns into a mediocre system.

The coding loop cares how fast the first token shows up: a person staring at a spinner goes and reads something else. The nightly job doesn't care about first-token latency; it cares whether every document got processed by morning. The chat sits in between and mostly wants to not feel broken. None of that needs a large GPU: queues, the API layer, an index for anything RAG-shaped (worth a look if you haven't built one: rag-that-retrieves), monitoring, and a small model doing routing or embeddings can live on the machine that's already running. Save the workstation for what needs it.

Baseline first, then argue with the numbers

Before touching anything, write down what you already have: one named model, one fixed set of prompts, the exact command that launches the server. Lose the model artifact or the prompt template and the comparison stops meaning anything.

Follow a request through the whole system, not the model alone. Time to first token covers loading and prompt processing, steady token rate covers decoding, and completion time is what the person on the other end feels. Add peak memory and queue delay when they change the decision, wall power if you're paying for it. For the nightly job, count completed valid jobs per hour, not tokens per second; a job that produces garbage quickly saved nothing. For the chat and the coding loop, count the slow waits, because those are what people remember.

I keep this in a plain log, one line per run: artifact, runtime, launch flags; workload and fixed input set; cold start, warm start, p50, p95; peak memory and wall energy; quality failures and abstentions noticed while reading the outputs; a decision, an owner, a retest date. That last field is the one people skip, and the one that matters most.

A measurement with no decision attached is trivia. A setting with no owner is folklore waiting to happen.

Soft failures don't trip any alarm

The failure mode I watch for: someone routes everything through the power-hungry box because it owns the main model, and it keeps working, because it still produces plausible text. Local inference fails in ways that never throw an error: a model partly spills onto CPU because you misjudged VRAM, a cache stops hitting, swap creeps up until the machine thrashes, a queue holds requests whose client disconnected an hour ago, a fallback path moves your data somewhere you didn't plan for. None of it trips a dashboard that only checks for a 200.

Watch the runtime logs and the OS counters while a test runs, not just the output. Change one variable at a time, unless you're deliberately comparing two full configurations. Run it more than once: a single fast pass tells you nothing about whether that speed is durable or just luck with a warm cache. Read the outputs, because speed is not proof the answer is still correct. If a change makes an important task worse, report that cost next to the gain.

Boring wins, and here's what it costs

Operational simplicity deserves its own line, not a footnote. An optimization needing manual repair after every driver update or model swap is not free; it's a small recurring tax. I'd rather run a configuration I can rebuild from a service file, a container definition, or a short script than a clever one only I understand during an outage. Skip the raw artifacts in the report if you like, but keep the hashes and the commands: an unrepeatable run doesn't count as a result.

The rule underneath all this: separate always-available coordination from expensive generation, mini PC owning the first, workstation owning only the second. Less exciting than the biggest accelerator you can afford, but it leaves a stack whose limits are visible, and a visible limit can be routed around, scheduled, or priced. An invisible one shows up as an unexplained wait and an emergency order two days later. Stop tuning once the workload hits its target with real headroom left; that headroom absorbs a longer document, one more user, or the next runtime release, not wasted idle capacity.

What I give up on purpose is speed on the rare request that needed the big box from the first millisecond, because routing through a coordinator adds a hop, one more thing that can be slow or down on a bad day. A mini PC tacking a few extra milliseconds onto a request beats a workstation quietly deciding, request by request, whether it feels like waking up.