Deduplicate identical in-flight LLM requests

Every team running a local model behind more than one caller has hit this: a burst of identical extraction jobs lands in the same second, the service spins up a separate generation for each one, and the box burns memory and queue slots on answers that were never going to differ. Nobody designed it that way. It's just default behavior when nothing told the handler two calls could be the same call.

The unit worth optimizing was never the model call, it's the completed task on a machine that stays responsive, and duplicate concurrent requests are the cheapest version of that: the same task, arriving twice, needing one answer. Coalescing beats caching here because the repeats land at the same moment, not minutes apart, the shape duplicate work takes when automation fires jobs in bursts.

Before touching a launch flag, write down which number is supposed to move: jobs accepted per hour, first-token latency, resident model capacity, energy per completed task, or corrections a human makes afterward. "Make it faster" doesn't tell you when you're done, or what you broke to get there.

What "identical" actually has to mean

Coalescing promises two callers the exact same output, so "same" has to be stricter than eyeballing the prompt text. Pin the model artifact, tokenizer, prompt template, runtime build, launch command, and every sampling setting: any one drifting means a caller is quietly served a different model than they asked for. Permission scope and freshness window belong in that key too:

Coalescing isn't semantic caching: it only merges identical prompts, never near-identical ones, which is what keeps it safe, worth reading against the case for semantic caching if you're tempted to loosen the match. Build the key against real inputs, the awkward ones automation actually sends, and test the case where the model is cold: a duplicate arriving mid-load should join that load, not trigger a second one.

One generation, several people waiting on it

The mechanism is simple: the first caller starts a real generation, later callers with a matching key attach as waiters, and none re-triggers the model. Independent cancellation has to survive that sharing: if caller B disconnects, caller A's job keeps running for the others attached; no one caller kills it for the group.

Measure it in phases, not one total; a single number hides which part of the pipeline actually helped:

• queue wait
• model load or activation
• prompt processing / prefill
• time to first token
• decode rate and completion time
• peak RAM, VRAM, power, and swap
• quality pass, retry, abstain, or repair

Raw token speed off that list is diagnostic, not the result you're chasing. What counts depends on who's calling: completed records per hour for automation, review and correction time for coding, p50 and p95 time-to-first-token for chat with the queue actually present. A layer that wins a short, warm, single-caller test can still lose across a real day once the model swaps and prompts get longer.

Coalescing also hands back headroom instead of spending it: every duplicate merged is a generation not run, memory the next request can use, queue capacity an interactive user doesn't wait behind, thermal margin left unspent.

Where two requests stop actually matching

The mistake that costs people is coalescing across tenants, or requests that look identical but carry different expectations: same prompt, different user, different permission scope, or a nondeterministic call with matching seeds by coincidence. Local runtimes make this easy to miss because they're stubborn about staying up, offloading layers, paging memory, eating a cache miss, queueing the request, or falling back to a generic kernel rather than failing loudly, quietly making a dedup layer look like it's working when it's just degrading gracefully for everyone attached.

Confirm it instead of trusting it: read the startup logs, check device placement, watch the operating-system counters, and verify the coalescing path is engaged for your real tensor shapes and context lengths, not a test script's, because every layer here is inspectable rather than a vendor claim. Change variables one at a time and keep outputs from every run, not just timings: quantization, context compression, and sampling can all make a shared generation faster while quietly changing what it says.

There's an operational cost too: upgrades, rollback, whether it shows up in observability, whether you could reproduce the server's behavior after a disk failure. A win depending on an undocumented patch or someone manually warming a cache every morning is a bad trade for a shared service, the kind of tradeoff a cost-focused architecture writeup covers in depth. Boring wins age well; clever ones need a maintainer who remembers why.

Put the retest condition in the same file as the result

When coalescing earns its place, write the conclusion next to the workload it was tested against, the date, and the reason it passed. Write the retest trigger beside it: a new model family, a driver update, longer contexts, another tenant, or a different traffic mix. Skip that and the benchmark quietly turns into folklore, repeated without anyone able to say when it was last true.

Leave slack after hitting the number you were chasing. Free memory absorbs prompt variance you didn't test for, spare queue capacity keeps an interactive user from waiting behind a batch job, thermal margin lets the box run the next hour the way it ran this one.

The rule I'd keep, if I had to keep exactly one: coalesce two requests only when their observable contract is provably identical, and the moment you can't state that proof in one sentence, run them separately.