Backpressure is kinder than an infinite inference queue

I've watched an inference queue do this before: it keeps accepting work long after it can't possibly serve it in time, and everyone downstream waits without knowing anything's wrong. A local box running a private chat, a coding loop, and a nightly extraction job took every request as it arrived, no limit, because memory was free and nobody had written the code path for saying no. One afternoon the extraction job dropped its batch in mid-conversation, the queue swallowed it, and ten minutes later somebody was staring at a spinner stuck behind an abandoned chat. Nothing crashed. Nothing logged an error. The system just stopped being useful for the one thing that needed to be fast.

one box, three different clocks

Generation capacity is finite, and waiting work goes stale fast. Skip the model name and the peak throughput figure; look instead at the work sitting on the other side of the API. A private chat used a couple of times an hour, a coding loop firing every few seconds, and a nightly extraction job chewing through documents can all live on the same GPU, each needing different treatment: a ten-minute-late interactive reply is worthless, a ten-minute-late batch job is nothing.

Before touching any configuration, get one baseline down: a named model, a fixed prompt set, the exact server command that produced it. Lose the model artifact or the prompt template and the comparison turns into noise later.

From there the fix is mechanical, not clever: a queue limit per request class, a retry hint instead of silence when a class is full, cancellation for work tied to a dead connection, batch jobs deferred behind interactive traffic. Run Ollama as the front end and this logic sits ahead of it, not inside it.

what the logbook actually has to say

Whatever you measure has to follow the request through the whole path, not just the easy part. Time to first token catches loading and prompt processing; steady token rate describes decoding; completion time is what the user experiences, waiting included. Add peak memory, queue delay, and wall power when they'd matter. For batch work, count completed valid jobs per hour, not requests attempted; a job that fails halfway isn't throughput. I keep the log line boring on purpose:

model + runtime + launch flags
workload class + fixed input set
cold start, warm start, p50, p95
peak memory, wall power
quality failures and abstentions
decision, owner, retest date

That last field is the one people skip, and the one that matters most. A number without an owner turns into folklore within a month; one with a retest date gets revisited when the runtime changes underneath you.

nothing here will throw an exception

The easy mistake is accepting every request because memory happens to be free; the system keeps producing plausible text right up until it doesn't. Local inference fails quietly, not loudly:

• a model partly offloads to CPU
• a cache misses and every request pays the cold cost again
• swap creeps up and the box thrashes instead of computes
• a queue keeps holding requests from connections that already closed
• a fallback path quietly changes the privacy boundary, routing a request elsewhere

None of that raises an error. You have to go looking: watch runtime logs and OS counters while a test runs, one variable at a time, unless you're deliberately comparing whole configurations. Run it more than once, since one good result can be luck rather than a real improvement, and read the output instead of trusting speed as proof quality held. A change that makes an important task worse should report that cost beside the gain.

Simplicity deserves its own line item: an optimization that needs manual repair after every driver or model update isn't free just because it was fast on measurement day. A boring setup reproducing from a service file, a container definition, or a short script beats a clever one only its builder can restart. Keep raw artifacts out of the report; keep the hashes and exact commands so a rerun months later still means the same thing.

reject fast, then leave yourself slack

The rule I'd keep: reject early whenever the service can't meet a request's deadline, instead of letting it sit in a queue pretending otherwise. Duller than picking the biggest model or the newest card, but it leaves you with a stack whose limits you can see. A visible limit gives you options: route around it, push the work to a quieter time, or price it for whoever wants more capacity. An invisible one gives you none of that; it just shows up as waiting nobody can explain, and eventually an emergency order for more hardware.

Stop tuning once the workload hits its quality and latency target with room left over. That margin isn't waste; it's what absorbs a longer document, one more concurrent user, a hot afternoon, or whatever the next runtime release changes underneath you. You don't need the queueing sophistication of a team running vLLM across a rack of H100s to get this right on one box, just an honest sense of what's allowed to wait.

None of this helps if you can't tell interactive traffic from batch traffic at the point the queue decides, and on most home setups nobody built that classifier: requests arrive shaped identically and get sorted by vibes. I've shipped exactly that setup myself: backpressure on paper, one undifferentiated pile of requests in practice.