Prevent retry storms on a local model server

Retries are what actually take down a shared local model server, not slow inference. That's a little unfair to the retries, which are usually just doing exactly what someone configured them to do: resubmit the same expensive prompt the moment a response feels slow.

Every layer thinks it's the only one retrying

Here's the shape of it: a shared endpoint sits behind application clients and a batch of scheduled workers, and each one carries its own retry policy because whoever wrote it wanted resilient code. The client backs off and tries again. The scheduler's queue does the same on its own clock, unaware the client already resubmitted. The load balancer adds a third layer of nearly identical short delays. None of these decisions is wrong alone. Stacked, they turn one slow response into several times the original demand, aimed at the already-bottlenecked server.

Local runtimes make this worse by being too good at coping, the same instinct that makes Ollama pleasant day to day: they offload layers, page memory, eat a cache miss, or drop to a slower kernel rather than fail outright. Cool. That resilience hides a retry storm until the queue is unmanageable.

The fix isn't clever, it's assigning ownership: one layer, and only one, decides when to retry, the kind of rule I'd bake into a shared box like running vLLM for a team:

retry:
  owner: gateway        # only this layer retries
  policy: bounded_exponential_backoff + full_jitter
  idempotency_key: request_id
downstream:
  app_client_retries: false
  scheduler_retries: false

That's bounded exponential backoff with jitter, a retry hint the caller respects, a stable request identity so a resubmission can't double as a second job, and admission control that rejects new work past a queue threshold. Every other layer shuts up and waits, because the moment two layers both think they're the safety net, neither one is.

Proving the fix instead of trusting it

None of this is worth doing on faith. Pin the model artifact, tokenizer, prompt template, runtime build, and sampling settings before touching the retry policy, then run the same small set of real inputs, including the ones that used to trigger retries, cold once and warm a few times after. Time queue wait separately from prefill, first-token latency, and decode: a fix that shaves the average while leaving the tail untouched hasn't stopped the storm, it's made it quieter to watch. Raw tokens per second tells you almost nothing here, what matters is completed requests per hour under the new admission control and the p95 first-token latency for whoever is actually waiting.

Because a retry fix tends to travel bundled with other changes, quantization, a shorter context, a different sampling temperature, run the quality check against every candidate configuration, the fast one included. A server that answers instantly and wrong hasn't been fixed, it's been rebranded.

Write the new policy down next to the reason for it and the workload it was tested against, and note what should trigger a retest: a new model family, a driver update, longer contexts, more concurrent users than tried. Leave real headroom in the queue too, spare capacity a batch job can't eat when someone's waiting on a chat reply.

None of this matters if you don't control the client code, and in most shared setups you only own two of the three layers that retry. Fix the two you can reach and hope the third one behaves.