Do not spend inference on deterministic work

So no, the fix isn't a faster model. Half of what looks like an inference problem is a step whose answer was already fixed before the prompt got built, routed through the model anyway because that was the easiest line of code to write.

Run a document pipeline for a while (extraction, normalization, classification) and the two kinds of work separate on their own. Extraction and classification carry real ambiguity: input varies, categories overlap, a human would hesitate too. Normalization mostly doesn't: dates, casing, whitespace, dedup, one right answer, a regex already knows it cold. I use one like that as my reference case, ordinary enough that the pattern generalizes to whatever your workload is: write down what the job is and what a passing result looks like before touching a single setting. That habit matters more once you're chaining calls together, the whole premise behind agent-architecture.

mark the step before you tune the model

Walk the pipeline and mark each step deterministic or judgmental before touching any setting. Deterministic steps, validation, formatting, dedup, get pulled into ordinary code, no model involved. What's left is the only thing worth benchmarking. Test on the same fixed inputs every time and log the launch command beside the result; you won't remember it later. One warm run tells you nothing: include a cold start if a human will hit one, run long enough to expose thermal throttling or queueing, and keep output quality next to the timing.

A speedup that changes the answer isn't a faster version of the old system. It's a different one.

Record only numbers that could flip the decision: time to first token, prompt-processing speed, generation speed, peak memory, wall power where relevant, whether the task succeeded. Medians describe a normal afternoon. A p95 describes the pause that makes someone close the tab.

when the load screen lies to you

The trap I see most: handing a model sorting, deduplication, arithmetic, reformatting of data that already has structure. It runs. It produces a plausible answer. Neither is a result. Local inference happily starts on a configuration that's quietly wrong: check runtime logs and OS metrics, not the flag you set. When two runs disagree, change one variable and say why you expected that to move the number, or the benchmark is just anecdotes.

There's a bill due later, too. A fragile five-percent win evaporates the moment a driver or runtime updates, and I've stopped chasing those. What I keep is small: one script, a handful of representative prompts, a plain-text results file I can diff. Enough to retest after an upgrade without a monitoring platform for one workstation, which I wouldn't bother building.

The rule that's stuck: reserve the model for decisions you can't cheaply write as rules, test the smallest change first, and stop tuning once the workflow clears its latency and quality bar, near the case in 99-percent-cost-architecture. What I still haven't figured out is where that boundary itself starts moving: thresholds drift, edge cases pile up, and a step that was safely deterministic when you built it quietly needs judgment again by the time anyone notices.