Prompt ingestion can dominate local latency

So you shave a real chunk off decode speed and the user still waits about as long as before. That's the tell. Somewhere between the request landing and the first token, something ate the budget, and it wasn't generation.

Take a local coding assistant that resends repository context and history each turn: the model reads every input token before writing one output token. That's prefill, and it doesn't care how fast decode is. Push context past a few thousand tokens and prefill eats the interactive budget while tokens-per-second still looks great.

Decode speed is a real number. It just measures the wrong half of the request.

the meter you're actually running against

Before you touch a launch flag or price a bigger card, decide what number you're moving. "Make it faster" doesn't say when to stop, and it won't catch a change that speeds the fast path while breaking the answer underneath. First-token latency. Accepted jobs per hour. Resident model capacity. Energy per finished task. Fewer human corrections. Pick one and hold every comparison against it, not a vibe.

pin it down before you turn a knob

Pin the model artifact, tokenizer, prompt template, runtime build, launch command, and sampling settings, then write it down somewhere you'll reread. Pull inputs from the real workload, including a couple of awkward cases: odd encodings, absurd context lengths. Run the cold path once if users hit it, then run warm long enough for cache effects, queueing, memory pressure, and thermal behavior to surface.

split the clock into its actual pieces

Split the clock instead of trusting one total: queue wait, model load, prompt processing (the prefill phase), time to first token, decode rate, completion time, peak RAM, VRAM, power draw, and swap, plus whether the output passed the quality check, needed a retry, or got abandoned. That figure is a diagnostic, not a verdict. The real result depends on the job: valid completed records per hour for automation, review time folded in for a coding assistant since a fast wrong answer isn't cheap, p50 and p95 first-token latency at realistic conversation lengths for chat, worth reading against streaming-latency-architecture once you're deciding how a slow prefill gets shown to the user mid-request.

the failure that hides behind a working system

The recurring mistake is tuning decode speed while the user waits inside prompt evaluation. Local runtimes hide this: they offload layers, page memory, miss caches, queue requests, or fall back to a generic kernel, and still answer. That resilience is a disguise: read the startup logs, check device placement, watch the OS counters, confirm the active optimization applies to the tensor shapes and context length you're running.

Change one variable at a time unless comparing two whole systems, and keep the outputs, not just metrics. Quantization, context compression, sampling, model swaps: any of them can make the answer worse while making it faster. Run the quality gate on every candidate. Shorter answer, check it's complete; faster extractor, count valid records, not parseable braces.

There's a cost beside raw performance: startup, upgrades, observability, rollback, rebuilding the server after a disk failure. A five-percent win that depends on an undocumented patch or manual cache-warming every morning is a bad trade for anything shared. Boring configurations age well.

write an expiration date on the number

I wouldn't bother squeezing more out of decode when prefill already eats most of the clock: that's optimizing the wrong half of the bill. The rule that's held up for me is cutting repeated input work before buying faster generation hardware, since prefill is a tax paid every turn no matter how good the GPU decodes. Write the conclusion into your result file with the workload, date, and reason, plus what should trigger a retest: a new model family, a driver update, longer contexts, a different traffic mix.

Leave headroom once you hit the target. Free memory absorbs prompt variance, spare queue capacity protects interactive users from a batch job, and thermal margin keeps things steady at hour six as at minute one. The goal is finishing real work predictably on the cheapest, least fragile setup that clears the bar, not maxing every resource, and every layer here is inspectable, so the number never has to be a vendor's claim.

The thing I'd check next: pull up your own logs and see what fraction of wall-clock time each request spends in prefill before buying anything bigger.