Use a cheap first pass and an expensive second pass

My rule, stated before I justify it: the small model touches the raw material, the big model only touches conclusions, and it doesn't get a turn until the small model can't tell yes from maybe. Everything below explains why that rule survived contact with a real workstation instead of dying in a spreadsheet.

Most workloads split into two kinds of work whether anyone designed them that way or not: a routine stretch where the model reads and structures, and a smaller stretch where it decides something. The mistake I keep making is optimizing the visible number, usually raw throughput, while the part that's actually wrong sits untouched in the reasoning half.

Sizing the job before you touch a knob

The setup I use here is unglamorous on purpose: a local 7B model doing the grunt work, paired with a stronger model behind it, local or hosted. I'm not claiming it's optimal for your hardware, just ordinary enough to reason about honestly. Before touching a setting I write down the job, chat, code completion, extraction, an overnight batch, and decide what a good result looks like before I decide what a fast one looks like.

The small model structures evidence, pulls fields, summarizes, classifies, and validates its own output before anything escalates. Only ambiguous cases go up the chain, the point of squeezing the local tier before you pay for anything bigger. I run every test against the same fixed inputs and save the launch command with the result: a config you can't reproduce isn't a result, it's a memory. I include a cold start whenever a real person would hit one, and repeat until heat or queueing has a chance to show up. Task success rides along with every timing number: a speedup that changes the answer isn't a faster version of the old system, it's a new one wearing the old name.

What gets recorded stays narrow: time to first token, prompt speed, generation speed, peak memory, wall power when it matters, whether the task passed. Medians describe the ordinary run; the slow end of the distribution is where the irritating pauses live, and an average erases them. Setup friction doesn't fit in a number, so it goes beside the numbers, because it decides whether anyone still uses the thing once the novelty wears off.

Don't let the cheap model finish your sentence

The trap: the small model writes something that reads like a finished conclusion, confident, well-punctuated, wrong in a way that's hard to spot, and the expensive model inherits that framing instead of the raw evidence under it. Once a paragraph sounds settled, even a much better model tends to edit it rather than re-derive it. Real routing by uncertainty means the small model hands up fragments and flags, not opinions dressed as findings.

Local inference is full of configurations that technically run. That's a low bar. "It loaded" is not a performance result. "The answer looked fine" tells you nothing beyond how someone felt reading it. What tells you something real is the runtime logs and the OS's own metrics, not whatever flag you asked the process to respect, since plenty of runtimes accept a setting and quietly ignore half of it. When two runs disagree, change one variable and say what mechanism you expect to move.

What has to survive the next driver update

There's a maintenance cost here, easy to ignore while pleased with a win. A fragile five-percent gain evaporates the moment a model, driver, or runtime changes underneath it, and workstations get those changes constantly, invited or not. What survives is boring on purpose: a small script, a handful of representative prompts, a plain-text file with the last result. That's enough to retest an upgrade in five minutes, not build an observability platform for a machine under someone's desk. I've watched people build elaborate tracking for one recurring job on one workstation, and the tracking outlived the workload.

Leave real headroom instead of running flat out; spare capacity absorbs the longer prompt, the background process, next month's model. The rule doesn't change: the cheap model sees everything first, the expensive one only sees what the cheap one couldn't settle.