Cap output length before tuning the model

Cutting the size of the answer beats almost every other latency knob on a local model, and I'd stake real hours on that. It gets weaker fast, though: it only holds once you already know what a good answer looks like, and most setups I've poked at never wrote it down.

The rig I keep coming back to runs a local assistant for code review and short operational summaries: nothing exotic, just a box someone sits next to every day. Before touching quantization or batch size, name the job, interactive chat, autocomplete, document extraction, or an overnight batch nobody watches, each with its own definition of good, or you'll end up shaving milliseconds off a number nobody asked about.

Shrink the ask, then start the stopwatch

The actual test is boring on purpose: ask for a decision with its evidence and a next action, each under an explicit size limit, expanding only on request. Run the same inputs every time and save the launch command with the result. One warm run tells you nothing. Include a cold start if a human will hit one, run long enough to expose thermal throttling or a queue backing up, and check the output stayed correct: a faster wrong answer just swapped the whole system out from under you.

Log only numbers that could change the decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power when relevant, and task success. Medians describe a normal Tuesday; the p95 catches the pause that makes a colleague ask if the machine froze. Notes on noise and responsiveness, plus setup friction, belong right next to those numbers.

task:      one named, repeatable job
inputs:    fixed, versioned, same every run
quality:   pass / fail / abstain
latency:   cold, warm, p95
verdict:   keep, revert, retest

The kernel wasn't the problem, the tutorial was

Here's the trap I see most: a weekend on quantization formats and kernel flags while every response still ships with a tutorial nobody asked for. Local inference has a big ecosystem of configurations that technically run, and loading successfully isn't a performance result, any more than a fine-looking answer counts as one. Pull the runtime logs and OS metrics instead of trusting the flag you just set. Change one variable per run, say what mechanism you expect to move, or the folder fills with anecdotes.

Part of that is perceptual: a fast first token changes how the response feels even when total time is identical, which is really a question about where those milliseconds go, not about the cap.

There's also a shelf life on cleverness: a fragile five-percent win disappears the next time the model, driver, or runtime gets bumped, and nobody notices until it just feels slower. Keep a small script, a few sample prompts, and a plain text result file, enough to retest after an upgrade without a monitoring stack.

Where I actually draw the line (tokens)

My rule is simple: brevity is part of the interface contract, not a fix bolted on once things feel slow. Test the smallest plausible change first. Keep some headroom rather than tuning to the wire: that spare capacity absorbs a longer prompt, a background app stealing memory, or next month's model. Stop tuning once the workflow hits its latency and quality target, even if the profiler insists otherwise.

Local models reward that restraint more than raw hardware does. Clear boundaries, honest measurement, fewer wasted tokens usually produce the first real improvement, before anyone touches a kernel. What matters in the end is a setup you trust enough to leave alone.

None of this saves you from the case where the honest answer needs four paragraphs and you've already capped it at one. You get the short version, feel good about the number, then burn two follow-ups pulling out the rest anyway. The wall clock doesn't care how proud you were of the cap.