When a distilled model is the better local model

Say you've got one GPU, a daily coding habit, and two candidates loaded up: a 32B general model that handles almost anything passably, and an 8B model distilled from something much bigger, tuned narrow on the kind of work you actually throw at it. The distilled model stays resident between sessions. The big one gets swapped out and pays a cold-load tax every time you come back to it after a meeting.

Here's the part that trips people up: the small student can genuinely beat the big general model, but only on the narrow behavior it inherited from its teacher. Step outside that behavior and it falls apart faster than the general model would. So before you touch a launch flag or add a GPU, write down, specifically, what has to get better. First-token latency. Accepted jobs per hour. How many models you can keep resident at once, energy per completed task, or just fewer corrections you make by hand afterward. "Make it faster" doesn't tell you when you're done, and it won't stop you from shipping a change that makes the answers worse while the clock looks better.

make the student prove it on your workload

Pin the exact things that could quietly change under you: the model artifact and tokenizer, the prompt template, which runtime build you're on, the launch command, the sampling settings. Write it down somewhere you'll look at again. Pull a small set of inputs from the real workload, not a demo set, and make sure it includes the awkward cases, the long file, the ambiguous ticket, the prompt that made the last model hedge. If your users actually hit a cold-storage, unloaded-model path, run that once. Then run warm, repeatedly, long enough for cache effects, queue behavior, memory pressure, and thermal throttling to show up, because none of that shows up in the first thirty seconds.

Test the specific distilled model you're considering on your actual tasks, and compare failure types, not the reputation of the model family it came from. A best-local-llms-for-coding leaderboard tells you what a model can do in general. It doesn't tell you what your 8B student does with your codebase's weird import graph. Record phase-level timings instead of one grand total:

• queue wait
• model load or activation
• prompt processing and prefill
• time to first token
• decode rate and time to completion
• peak RAM, VRAM, power draw, and swap
• quality pass, retry, abstain, or repair

Raw tokens per second is diagnostic evidence, not the product result. For batch automation, count valid completed records per hour, not raw tokens. For coding work, fold in your own review and correction time, since a fast wrong answer costs more than a slow right one. For chat, check p50 and p95 latency to first token across conversations that actually run long, not just the opening exchange. A configuration that wins a short, warm, batch-of-one benchmark can still lose the whole day once the model has to swap and the context gets long.

the resilience that hides a bad config

The easy mistake is assuming distillation carries over every capability of the teacher just because the outputs look similar on a couple of prompts. Local runtimes make this worse because they're stubbornly, admirably willing to keep working no matter what you throw at them: they'll offload layers to the CPU, page memory, eat cache misses, queue requests behind each other, or quietly fall back to a generic kernel instead of the one you meant to use. None of that throws an error, and all of it hides a degraded configuration behind numbers that still look plausible. Read the startup logs. Check device placement. Watch the OS-level counters while it runs. Confirm the optimization you asked for is actually active for the tensor shapes and context length you're using, not the ones in somebody else's benchmark.

Change one variable at a time, unless you're deliberately comparing two complete systems against each other. Repeat every sample and save the actual outputs, not just the metrics, because quantization, context compression, sampling tweaks, and model swaps can all change the answer while making the run look faster. Run the quality gate on every candidate configuration, no exceptions. If an answer comes back shorter, check that it's still complete. If an extractor got faster, count the valid records it produced, not the braces that happened to parse.

Operational cost sits right next to the latency number, and it's easy to forget about while you're staring at the latter. Note what the change does to startup time, upgrades, observability, rollback, and your ability to rebuild the box from scratch after a disk failure. A five-percent speed gain that depends on an undocumented patch, or someone manually warming the cache every morning, is a bad trade for anything more than a single-user toy. Boring holds up. Clever pages you at 2 a.m.

The rule worth writing down: use the student where its evaluated specialty matches the workload in front of you, and nowhere else. Put that conclusion in the result file next to the workload, the date, and the reason, along with the condition that should trigger a retest, a new model family, a driver update, longer contexts, a different user population, a shifted traffic mix. Skip that part and the benchmark numbers slowly calcify into folklore nobody double-checks.

Leave headroom once you hit the target: spare memory to absorb prompt lengths you didn't test for, spare queue capacity so an interactive user never waits behind a batch job, and thermal and power margin so the machine performs the same at hour six as at minute one. That's efficiency at the resident-model layer: useful work finishing on schedule, on the cheapest and least fragile setup that clears the bar, with the card nowhere near pinned at a hundred percent.

The part of this worth having on hardware you actually own is that every layer of it is inspectable, so you're never just taking someone's word for the number. What I'd check next, on that box: rerun the warm suite the moment your context lengths grow past what you tested, or the moment a driver updates, and confirm the student's evaluated specialty still holds before you trust the old result again.