Set different timeouts for queue, first token, and stream

A request lands on a shared local model behind a streaming endpoint. It queues behind three others, waits for the model to load, streams tokens, then finishes, and your client has one timeout wrapping all of it. When it fires you learn nothing except that something was slow. Was the queue backed up, was the model still loading, or did generation stall near the end? One number can't tell you which, because each phase has its own duration and correct recovery.

Three clocks, one timeout

Queue wait, model load, prefill, and decode are different kinds of wait, and treating them as one collapses the diagnosis along with the deadline. Pin the model artifact, tokenizer, prompt template, runtime build, and sampling settings, then test a small set of real inputs, awkward ones included, cold once and warm repeatedly. What you're building is a set of independent deadlines, each with its own response:

• queue wait: fail fast, it should never have been accepted
• model load: wait it out or reroute, not the request's fault
• prefill and first token: a generous ceiling, prompt length varies
• decode: watch the rate, not just the total, a mid-stream stall isn't a slow start

Raw tokens per second is diagnostic, not the product result: track completed work per hour, or first-token p50 and p95 under realistic conversation lengths. A gateway in front of the model is the natural place to own these deadlines, since a client alone can't tell which failure it hit.

The runtime won't tell you it's degrading

The mistake I see most is retrying a request that's still generating, doubling the load on a server already struggling. Local runtimes are stubbornly cooperative: they offload layers, page memory, miss cache, queue silently, or fall back to a generic kernel rather than fail outright. That's good manners and a bad diagnostic signal: a degraded configuration keeps answering, just slower, so check the logs, device placement, and OS counters before trusting an optimization is live for this context length. Change one variable at a time, and don't trust a speed win until the quality gate passes it, since a shorter answer might just be incomplete. Retry only the phases where retrying is safe. That's the whole rule.

What margin costs you

I leave the system short of its ceiling on purpose. Spare queue capacity means an interactive user doesn't wait behind someone's batch job. Free memory absorbs a prompt longer than the test set. Thermal headroom means the fifth hour looks like the first. A config that shaves latency with an undocumented flag or a manual warm-up ritual just moves the cost to on-call. None of it is free: it's throughput I could claim and don't, capacity left on the table, traded for a server that degrades predictably instead of falling over the moment three requests land at once.