Build a boring local model router

So the router doesn't get to decide anything on its own. It executes a rule I wrote down before the first prompt went through it. Three models sit on the machine day to day: something fast for general chat, a coder model, and a slow large model I wake up only when the first two shrug. Nothing exotic, nothing that claims to generalize to your rig. A decision I can't observe is a decision I can't trust, no matter how good the demo looked an hour ago.

Name the job before you touch a knob

Before any model sees a request, the request needs a label. Interactive chat is not code completion. Document extraction is not an overnight batch job, and treating them the same is how routing turns into a lottery. Each label carries a definition of what good means, plus a hard limit: a context ceiling, a latency ceiling, sometimes both. Break the ceiling and the request falls through to the next tier automatically, and every fallback gets logged. Once a week I read that log for the same shape misrouting twice. Fifteen minutes catches more than any dashboard I've built.

Cold starts and the boring repeat

Testing this well is duller than it sounds, on purpose. Same inputs every run, versioned, so nobody argues later about what executed. One warm sample tells you almost nothing, because the first request of the morning is never the warm one. So a cold start goes in the mix, and the run repeats long enough for heat throttling or queueing to show up. I keep only numbers that could change the decision: time to first token, generation speed once rolling, peak memory, wall power when it's relevant, and whether the output passed. Medians describe an ordinary Tuesday. The slow percentile describes the moment a reviewer decides the tool is unreliable and quits opening it.

A speedup that changes the answer isn't a faster version of the old system. It's a different one wearing the old name.

The router that hires another router

The trap I keep seeing: someone points a second LLM at deciding which model should handle the first LLM's request. Now there's the original cost, a routing cost on top, and a new failure mode an eyeball won't catch. Local inference is full of setups that technically run. Loaded is not a performance result. "The answer looked fine" is an opinion offered at eleven at night by someone who wants to close the laptop, not an evaluation. I trust the runtime log and the operating-system metrics instead, not the flag I asked for, because that flag isn't always the flag that ran. When two runs disagree, I change one variable and write down the mechanism I expect first. Skip that and the benchmark is just a folder of anecdotes.

A five-percent win with a shelf life

There's a maintenance bill nobody puts on the spreadsheet. A fragile five-percent gain survives exactly until the next driver update or the next coder model gets swapped in, then it evaporates and nobody notices for a month. I'm not building an observability platform for one workstation against that. A short script, a handful of prompts that represent the workload, and a plain-text results file is enough to retest after every upgrade. For the fuller case on why headroom matters more than a maxed-out card, see hardware for local LLMs.

Rules first, headroom always

My rule fits in one breath: keep the plain rule until real traffic proves an ambiguity it can't cover, then go looking for something cleverer. Test the smallest change that could plausibly work first. Stop tuning the moment the workflow clears its latency and quality bar, not when the number stops moving, because those are different finish lines. Leave capacity on the table on purpose too: a longer prompt than usual, a background process you forgot about, next month's bigger model, all of it needs somewhere to land. For the fuller argument on going past this, building an autorouter covers it; I just haven't hit the ceiling that would force my hand.

The tradeoff I've made on purpose: this setup will never find the clever gain a proper learned router might eventually earn by understanding a request in ways a hard limit can't. I gave that up deliberately. What I get instead is a system where every misroute has a reason I can point to on the log, and that's worth more to me at 2am than a number I can't explain.