Include human review in LLM efficiency math

A 14B-class GGUF on a 12 GB card will hand you a finished-looking answer before you've finished reading the prompt, whether or not it's finished. The card isn't struggling. The cost lands downstream, in the chair, where someone reads the output and decides whether to trust it.

That gap between generation speed and answer quality is where the real price of local inference lives. In coding, extraction, and content workflows, review time routinely outruns inference time. Cut decode latency in half and you can still lose the day, because typing speed was never the bottleneck.

The unit worth measuring isn't the model. It's a finished task on a machine still doing its other jobs: evidence in, answer out, something checks the answer. Optimizations that look great alone tend to disappear once you put them back in that loop.

decide what "faster" has to mean first

Before touching a launch flag or swapping hardware, write down what actually has to improve:

• first-token latency
• accepted jobs per hour
• resident model capacity, how many jobs run alongside this one
• energy per completed task
• fewer human corrections per batch

"Make it faster" doesn't point at any of these, so it can't tell you when to stop. Pick one, maybe two, and let everything else be a constraint you're not allowed to break.

freeze everything except the one variable you're testing

A baseline only means something if you can reproduce it: pin the model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Use real workload inputs, awkward ones included. Run once cold if users hit that path, then warm, long enough for cache effects, queueing, memory pressure, and thermal throttling to surface.

What you're really after is touch time: how long a human spends on the output, what correction categories come up, how many cases get waved through with no review at all. Record it by phase, not one total:

queue wait
model load or activation
prompt processing / prefill
time to first token
decode rate and completion time
peak RAM, VRAM, power, and swap
quality pass, retry, abstain, or repair

Raw token speed is a diagnostic, not the result. For automation, count valid completed records per hour. For coding, fold review and correction time into the number, or it's lying to you. For chat, watch p50 and p95 first-token latency across realistic conversation length. A setup that wins a short batch-of-one run can still lose once model swaps and long contexts show up in production.

local runtimes fail quietly, not loudly

The mistake I see most is claiming a token-cost win while quietly shifting labor onto whoever reads the output. Local runtimes are almost too accommodating: they offload layers, page memory, eat a cache miss, queue a request, fall back to a generic kernel rather than error out. Good survival trait, bad for measurement: a badly configured run and a well-configured one look nearly identical until you check.

So check. Read the startup logs, inspect device placement, watch the OS-level counters for memory and swap. Confirm the optimization you enabled is active for the tensor shapes and context length you're actually using.

Change one variable at a time, unless the run explicitly compares two complete systems. Keep outputs, not just metrics: quantization, context compression, sampling, and model swaps can shift the answer while the run looks faster. Run the quality gate against every candidate. Shorter answer, check it's complete. Faster extractor, count valid records, not braces that parse.

the invoice benchmarks don't itemize

Performance numbers travel alone too often. Put operational cost beside them: startup, upgrades, observability, rollback, rebuilding the server after a disk dies. A five-percent gain resting on an undocumented patch, or someone warming the model each morning by hand, is a bad trade for anything shared. Boring configurations age well. Clever ones need a maintainer who remembers why.

write down the retest trigger, not just the win

Once you have a result, pick the workflow that reduces verified completion time and write that down with the workload, date, and reasoning. Then write the condition that forces a retest: new model family, driver update, longer contexts, different user population, shifted traffic mix. Skip that and the benchmark quietly turns into folklore, repeated long after the hardware or workload moved on. That's the same discipline that keeps a local-first cascade trustworthy instead of a one-time demo.

Leave headroom once you clear the target, rather than packing the system to its limit. Free memory absorbs prompt-length variance. Spare queue capacity keeps an interactive user from feeling a batch job land on them. Thermal margin keeps the box behaving the same late in a long run as it did at the start. Efficiency here isn't maxing out every resource, it's the cheapest, least fragile system that clears the bar consistently.

I don't bother squeezing more tokens per second out of a config once decode is already past reading speed. No amount of kernel tuning fixes a slow reviewer.

What I still don't have a clean answer for is delayed failure: a record that clears review today and turns out wrong well after the fact, once everyone downstream has already treated it as accepted.