Teach the local workflow to accept “I do not know”

So here's the piece people skip when they bolt a small model onto a pipeline: giving it permission to say no.

A small model running locally will answer almost anything put in front of it, whether or not it knows. That's the trap. Wire it into a classifier or extractor with business rules downstream and confidence has nothing to do with correctness. Score only what it answers and production quietly forces guesses through the gaps.

The abstain output is the feature

Before touching a launch flag or swapping hardware, decide what you're optimizing for: first-token latency, accepted jobs per hour, resident model capacity, or fewer manual corrections. "Make it faster" doesn't say when you're done. Pick a target, define answerability, add an explicit abstain output, and calibrate it against real examples until it fires right. Report coverage next to accuracy, not accuracy alone. A model that's right most of the time it answers, but silently guessing on the rest, isn't strong. It's a landmine with good PR.

What has to stay fixed while you measure

Pin the model artifact, tokenizer, prompt template, runtime build, and sampling settings, then run it against real workload inputs. Go cold once if users hit that path, then warm long enough to expose cache effects and thermal drift. Tokens-per-second is a diagnostic, not the result. For automation, count valid records per hour. Coding cares about review time, chat about first-token latency. A config that wins a short warm run can lose the day once model swaps and prompts get long.

What the resilience is actually hiding

Local runtimes keep going no matter what: offload layers, swap pages, miss the cache, or fall back to a generic kernel rather than fail loudly, hiding a degraded config behind a normal response. Read the startup log and confirm the optimization is really active for the context in use. Change one variable at a time and save the raw outputs. Quantization, compression, a sampling tweak: any one can alter the answer while speed looks better. Rerun the quality gate on every candidate, the discipline you'd want from output validation.

Boring is the goal, not the compromise

Weigh operational cost alongside speed: startup, upgrades, rollback, recovery after a disk failure. A five-percent gain from an undocumented patch, or hand-warming it, is a bad trade for anything shared. Leave headroom past the target. Spare memory absorbs prompt variance, spare queue keeps a batch from starving the user beside it. Thermal margin is what keeps that headroom real once the machine has been running a while.

An abstain output that never fires is decoration, not calibration.

Write the conclusion into the result file with the workload, the date, the reason. Note what should trigger a retest, a new model family or a driver update, or contexts grown longer. Skip that and last quarter's numbers turn into folklore. Run locally and every layer is inspectable: none of it stays a vendor's claim.

What I'd check next: pull the abstain-rate logs and find the last request where the model should have declined and didn't.