Batching helps throughput and can ruin chat

Continuous batching will make your local server post better throughput numbers. It will also, without touching any other setting, add real dead air before your chat model's first token shows up, and that part usually surfaces from an annoyed coworker, not a benchmark.

one card, two jobs that don't share well

I run this against a single GPU serving two things: interactive chat, and a background extraction job chewing through documents. Continuous batching is built for the second job: it packs more requests into the same forward pass, so the tokens-per-second number goes up, sometimes by a lot. That same packing means your one chat request waits behind whatever else got batched in beside it. Whether that's good or bad depends on what you're optimizing for: total work done, or time to first word. Say which one, out loud, before touching a config. Write it down. It changes every default you'd reach for, including whether chat should skip the queue and get immediate execution instead.

the test I actually trust

The setup I trust is close to what a proper vllm-at-home build needs anyway: split traffic into classes, chat and batch, and cap how long anything sits in queue before it runs or gets bumped. Benchmark p50 and p95 for chat, not one throughput average for the whole server. Averages hide the queueing; the ninety-fifth percentile is where the pauses live. Run the same fixed prompts every time, log the launch command beside the result, and include a cold start if a real person will hit one, because a warm-cache number that never occurs in production tells you nothing. Run long enough that heat and queue buildup show up on their own. Keep answer quality in the same record as the speed: a faster setup that quietly changes what the model says isn't a faster version of the old system, it's a different one.

workload:      chat + background extraction, fixed set
queue cap:     max wait before reject or reroute
latency:       p50 and p95, chat class only
cold start:    included, never averaged away
quality gate:  pass / fail on the same prompts

where the queue quietly gets cut

The failure I keep running into is the boring one: a big offline batch job gets scheduled and holds a waiting chat request behind it, because nothing told the scheduler not to. The runtime reports the request as accepted whether or not a human waits an extra few seconds, so check the runtime logs and scheduler behavior, not the flag you set. Change one variable when a run behaves differently than the last, and have a mechanism in mind for why, or you're just collecting stories with numbers attached. A five-percent gain from some clever queueing trick evaporates the moment the model, driver, or runtime gets upgraded, and you won't notice until someone complains. A short script and a handful of representative prompts saved as plain text catches that; a single workstation doesn't need an observability platform.

My rule: reserve enough headroom that chat never waits behind the batch job, and let the batch job eat everything else, including whatever model you test next month. That's most of what decides whether streaming-latency-architecture even matters here. Stop tuning once you hit the latency target, and stop chasing the last ten percent. What I still haven't solved is how much headroom is enough on one consumer card once heat gets involved: thermal throttling on a long batch run shifts the answer partway through, after the queue cap is already set, and I don't have a clean way to catch that drift before a person notices the chat got slow.