Base or instruct model for a local application?

Base models still earn their disk space next to the instruct checkpoint. That claim holds only if what you're building sits closer to autocomplete or a fine-tuning target than a conversation. Chat wants a different starting point than raw continuation, and adaptation work wants something else again, not one checkpoint doing double duty as the default download.

Autocomplete doesn't want your chat template

The recurring mistake, and I've made it, is loading a base checkpoint, wrapping the prompt in chat markers it never saw, then blaming it for ignoring the system role. It isn't ignoring anything. It never learned a role exists; it just keeps going from wherever the text stops, and yours stopped mid-instruction. For structured completion, finishing a code block or a template, that's exactly what you want: plain continuation, no assistant persona involved. It's also the better start for fine-tuning it yourself: an instruct model already has opinions baked in, and unlearning them costs more than adapting one that never had any. Pick whichever objective sits closest to the product interaction, and write the reason down before you forget it.

Pin everything before you compare the two

Pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings for both variants first. Feed both the same slice of real workload, ugly inputs included. Run once cold, unloaded, if users will hit that path, then warm and repeated, long enough for cache effects, queueing, memory pressure, and thermal throttling to surface. Record the phases, not one total:

queue wait
model load / activation
prefill
time to first token
decode rate
peak RAM / VRAM / power
quality pass, retry, abstain, repair

Raw tokens per second is a diagnostic, not the answer. For automation, count valid completed records per hour. For coding, fold in review and correction time: faster is worthless if it moves work onto you. For chat, check p50 and p95 time-to-first-token across realistic conversation lengths, not one fresh turn. A setup that wins short and warm can still lose once model swaps and long prompts show up.

Fast is cheap, done and reproducible costs more

Local runtimes stay stubbornly willing to keep going: they offload layers, page memory, miss a cache, drop to a slower kernel, hand you an answer anyway. Good for uptime, bad for benchmarking: a degraded run looks fine until you check the startup log and confirm the optimization is active for the tested context length. Change one variable at a time unless you're comparing two whole systems, and keep the outputs, not just the metrics: quantization and context tricks can make a run faster and the answer worse at once, so the quality gate runs on every candidate, baseline included. There's operational cost too: a five percent speedup riding on an undocumented patch or unwritten warm-up ritual is a bad trade for anything you'll still run next year. Boring, documented configurations age better than clever ones. Leave headroom once you hit the target: spare memory absorbs prompt variance and spare queue capacity keeps a batch job from starving an interactive user. Thermal margin covers the hour the machine actually gets pushed hard. Write the choice down, base or instruct, with workload, date, and reason, and note the retest trigger: new model family, driver update, longer contexts. Skip that and the numbers quietly rot into folklore.

What I haven't solved is the workload right on that boundary: a retrieval pipeline that pastes unpredictable context into the prompt and then wants instruction-following at the end of it anyway. I've run both checkpoints against it. Neither stays right for long, and I still don't have a good rule for when to give up and switch.