A practical local-first, cloud-second policy

My daily driver is a local model running on hardware I own, backed by exactly one hosted model I've approved as a fallback. Nothing else gets to see the traffic. That's the whole setup: not a lab rig, not a benchmark suite, just the machine on my desk and a single escape hatch I've decided to trust. The reason it works isn't the model choice. It's that I wrote down, before touching a single setting, what job this thing actually does for me: interactive chat, code completion, document extraction, or an overnight batch, and what a good answer to that job looks like. Skip that step and you'll spend a weekend tuning a number nobody asked about while the thing that actually slows you down sits there untouched.

The boundary has to be explicit or the whole arrangement is theater. If I can't tell you, before I hit enter, roughly where a given prompt is going to land, the system isn't routing, it's guessing with extra steps. So the rule I hold myself to is: classify sensitivity before capability. Decide first whether this is data allowed to leave the room, and only after that ask whether the local model can actually do the job. Ambiguous cases don't get auto-routed anywhere. They get a confirmation prompt, and I have to click it myself.

Where the line gets drawn, run by run

Testing this is less interesting than people expect, on purpose. Same inputs every run, saved alongside whatever the run produced, launch command included. One warm sample tells you nothing. You need a cold start too, because that's the state a human actually walks into half the time, and you need to run it long enough that heat and queueing show up if they're going to show up at all. Quality goes in the same record as the timing, always. A faster answer that also happens to be wrong isn't a quicker version of the same system; it's a different, worse one wearing the old name.

What actually earns a spot in the log:

• time to first token
• prompt-processing speed and generation speed, tracked separately
• peak memory, and wall power when it's relevant to where the box lives
• task success, judged the same way on every run

Medians describe the ordinary case. They also hide the thing that will actually annoy you day to day, which is the slow percentile: the run that stalls, the one where something else on the box decided to wake up right then. I keep a note next to the numbers about fan noise, how snappy the thing felt, how much friction there was getting to a clean run in the first place, because those costs are what decide whether a setup stays in use, and none of them show up in a median. This is close to the discipline behind building an autorouter: the classification step is only trustworthy once you can watch it decide, run after run, and see the same reasoning apply twice.

When the safety net turns into the leak

Here's the trap I actually walked into: a local run throws an error, and the system quietly falls back to the hosted model rather than surface a failure to the user. Which defeats the entire point of drawing the boundary in the first place. Local inference will happily hand you a dozen configurations that technically run without doing what you need, and loading without an error isn't a performance result any more than a plausible-looking answer is an evaluation. Check the runtime logs and the OS-level metrics; don't trust a flag just because you asked for it. When two runs disagree, change exactly one variable and be able to say why you expected the difference, or the whole exercise degrades into a pile of anecdotes with numbers stapled to them.

There's an upkeep tax too, and it's easy to underprice. A fragile five-percent win evaporates the moment a driver, a model, or the runtime underneath it updates, which on most machines is more often than anyone would like. So keep the footprint small: one script, a handful of prompts that actually represent what you do, a plain-text file of results. That's enough to retest after every upgrade without turning a single workstation into an observability project, a project I do not want and would not finish. The same instinct scales down cleanly into a local-first cascade: cheap and nearby first, escalate only when the workload has actually earned it.

The decision rule underneath all of it is short. Privacy constraints are hard rules, not preferences the router gets to weigh against convenience. Quality-driven escalation stays visible, so I always know when and why a request left the machine. Test the smallest plausible change first, leave headroom rather than tuning right up to the edge, and stop once the setup clears the latency and quality bar you wrote down at the start, not when the number simply stops improving, which is a different and much later stopping point. That spare capacity is what absorbs a longer prompt next week, a background process you forgot was running, or whatever model you decide to try next month. Hardware matters less than people assume. A clearly drawn boundary and a habit of measuring the same way every time gets you most of the improvement before you've spent a dollar on new silicon.

What I haven't solved is the confirmation step itself. Ambiguous uploads are supposed to stop and ask, but after enough repeated "yes, send this" clicks a person stops reading the prompt and just clicks through, which is exactly the case the confirmation exists to catch. I don't have a good answer for what replaces a habituated human at that checkpoint, and I'm mildly suspicious of anyone who claims they do.