Tune continuous batching for the users you have

The batch size that wins the best tokens-per-second number on a synthetic run is usually the batch size your users will hate. That's what happens when a scheduler is tuned for the whole queue instead of the person at its front. Continuous batching packs many in-flight requests onto one GPU, and that's where a vLLM-style server's throughput comes from. The same knobs that raise it decide how long a first token takes, and past some point the two goals pull apart.

Decide who the server is allowed to make wait

Maximizing aggregate throughput can produce queue and first-token delays nobody asked for. A server handling mixed context and output lengths has this tension built in: a big batch that saturates the GPU also makes a short prompt sit behind a dozen long ones. Run something like h100-vllm-team-serving, where several people share one card expecting live answers, and that tension is the whole job. Run a vllm-at-home box solo and it barely matters. Before touching a flag, write down what has to improve: first-token latency, jobs accepted per hour, models kept resident, energy per task, fewer human corrections. "Make it faster" doesn't tell you when to stop.

Time every stage, not just the demo run

Build a baseline you can rerun next month and get the same answer back. Pin the model artifact, tokenizer, prompt template, runtime, launch command, and sampling settings, then run real workload prompts through it. Do a cold run if users hit cold starts, then enough warm runs to expose cache behavior, queueing, and thermal throttling.

queue_wait
prefill_time
time_to_first_token
decode_rate
peak_vram
retry_or_abstain_rate

Raw decode speed is a diagnostic, not the product result: valid records per hour for extraction, review time plus generation time for coding, p50/p95 latency at realistic conversation lengths for chat. A setting that wins a short, warm run can lose badly once model swaps and long prompts show up. The quiet failure: benchmark short prompts, then ship those settings against traffic that looks nothing like them. Local runtimes keep going even when something's off, offloading layers, missing a cache, or falling back to a slower kernel instead of failing loudly, so a degraded config looks fine until someone checks.

Put a reason and an expiry date on every number you keep

Change one variable at a time unless comparing two whole setups, and keep the outputs from every run, not just the metrics: quantization and context tricks can shorten an answer without breaking it, or break it while looking shorter, and the quality gate must run on every candidate or it isn't a gate. Weigh the operational cost too: startup, observability, rollback, disk-failure recovery. A five-percent gain resting on a hand-applied patch and manual cache warming is a bad trade for a shared service. Boring configurations age well.

Pick scheduler parameters from your p95 latency target and completed useful work, not the biggest number on the chart, and write the reason down with the workload and date. Write the retest trigger too: new model family, driver update, longer contexts, one more team sharing the box. Skip that and last quarter's benchmark quietly becomes this quarter's folklore. Something like h100-overnight-batch, with nobody waiting on a first token, can pack the schedule tight; a shared card changes that.

So here's the tradeoff I take on purpose. I leave headroom after hitting the latency target: memory for prompt variance, queue slots for whoever just opened a chat, thermal margin for the run that goes long, instead of packing to whatever the benchmark says is possible. That headroom is throughput I'm choosing not to collect.