Small local models are excellent classifiers—after calibration

A 1B to 3B model has just enough capacity to sort a message into a handful of fixed buckets, and not much beyond that: the shape of the job, not a flaw in the model. Classification sits right in that sweet spot, narrow and cheap to run on hardware you already own. The catch isn't capability. It's that these models hand back a confident label for input they don't understand, and a plausible answer isn't a correct one.

My reference setup is deliberately unglamorous: a 1B-3B model triaging incoming messages on one ordinary machine, no lab conditions, no claim it generalizes to your box. Before touching a setting I write down the job itself: triage, nothing else. Then what counts as a good result, in words a colleague could check.

fixed labels, calibrated thresholds, an exit ramp

The method that answers whether this works: fix the label set, build a small balanced set of fixtures, calibrate a confidence threshold against them, and route anything under it to a stronger path instead of forcing a guess. That routing step is a local-first cascade by another name, and it's the piece most people skip. It shouldn't be: an abstain path is what keeps the small model honest.

Run the same inputs every time and save the launch command with the result. One warm run tells you nothing. Include a cold start and run long enough to expose thermal throttling or queueing under load.

Record only the numbers that could change your decision:

• time to first token
• prompt-processing speed
• generation speed
• peak memory
• wall power
• task success against the fixtures

Medians describe the typical run. A slow percentile describes the run that makes someone close the app. Keep both, and log the setup friction beside them: it decides whether anyone keeps using this.

labels:    fixed, small, versioned
fixtures:  balanced, held out from tuning
threshold: calibrated, not guessed
abstain:   routes out, never fails silently
retest:    after every model or driver change

the eval that flatters itself

The trap is specific and easy to miss: forcing every input into some class, then counting a plausible label as correct. It isn't. "It loaded" is not a performance result, and "the label looked reasonable" is not an evaluation, just a vibe. Pull the runtime logs and the OS-level memory and CPU numbers instead of trusting the flag you asked for. When two runs disagree, change one variable and say why you expect it to matter. Otherwise it isn't a benchmark, it's anecdotes with numbers stapled on.

There's a maintenance cost too, and it never makes the demo. A fragile five-percent win found by tuning against your own fixtures won't survive the next model release or driver update, especially once you're squeezing the local tier for everything it's got. Keep a small script, a handful of fixtures that resemble your real traffic, and a plain-text result file beside it. Nobody needs a dashboard for a workstation.

the rule I keep

Abstention counts as a successful outcome, not a parser error. Test the smallest plausible change first, leave headroom for the longer prompt and the background process you forgot about, and stop tuning the moment the workflow clears its latency and quality bar. Nothing after that earns more of your time.