Parallel prompt processing needs workload evidence

Bump the batch size on your ingestion job and the pile of documents clears noticeably faster. Leave the chat endpoint running on the same box and its replies start arriving a beat late, then late enough that someone notices. Nothing crashed. The GPU is busier than ever. The thing you promised users, a fast reply while they're waiting, quietly got worse.

Prefill and decode are not the same kind of work, and a scheduler that treats them as interchangeable will rob one to pay the other. Prefill chews through the whole prompt at once, so it's compute-bound and loves big batches. Decode emits one token per step, so it lives or dies on memory bandwidth and on how well the runtime interleaves requests. Crank parallelism on the prefill side without limiting how much of it can run before decode gets a turn, and the machine gets faster overall and worse for anyone waiting on a token right now.

None of that shows up if you only watch total throughput. It shows up once you time the two kinds of work separately, which is exactly where most people stop looking.

Time prefill and decode like they're different machines

Before touching a launch flag, decide what number you're trying to move. First-token latency for the chat path. Completed documents per hour for the batch path. Resident model capacity if more than one model shares the card. Energy per finished task if power is the real constraint. Fewer manual corrections if a human reviews the output. "Make it faster" doesn't say when to stop, or which side of the workload you can sacrifice.

Then pin everything else: model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Pull a small set of real inputs, not synthetic ones, and keep the awkward cases in there, the ten-page contract, the one-line question mid-batch. Run it cold once, model unloaded and cache cold, if users actually hit that path, then run it warm and repeatedly, long enough for cache effects, queueing, memory pressure, and thermal throttling to show up.

Then separate the traffic classes and load them together. Run chat requests alongside the batch job at realistic concurrency, and look at throughput and tail latency, not the average. Record the run phase by phase, not as one total:

• queue wait
• model load or activation
• prefill time
• time to first token
• decode rate and completion time
• peak RAM, VRAM, power, and swap
• quality pass, retry, abstain, or repair

Raw tokens per second is a useful diagnostic and a poor final answer. For the batch path, count valid completed records per hour, not tokens processed. For coding, count the review and correction time a human spends afterward. For chat, look at p50 and p95 first-token latency across realistic conversation lengths, not the best run recorded. Watch the trap that catches almost everyone: a config that wins a short, warm, single-request benchmark can lose badly once a second model swaps in or a long document lands in the queue.

The one big prefill batch that starves every decoder in the room

The mistake I see most often is the one from the opening. Someone queues one large prefill batch that gets first claim on the GPU, and every decode step for every other session waits behind it. It's an easy trap: the request looks efficient in isolation, short and high-throughput, done before you'd think to check on it. The requests it delayed don't show up in that log.

Local runtimes make this worse by being too resilient for their own good. When memory runs tight or a kernel doesn't match, they offload layers, page memory, miss a cache, queue the request, or drop to a generic kernel rather than fail loudly, keeping the system technically working while quietly changing what you're measuring. Read the startup log, check device placement, watch the operating-system counters instead of the runtime's own numbers, and confirm the optimization is actually active for today's tensor shapes and context lengths, not the ones you tested last month.

Change one variable at a time, unless you're deliberately comparing two whole systems. Save the outputs from every run, not just the metrics: quantization, context compression, sampling, and model swaps can all make a run faster while quietly changing the answer. If an answer got shorter, check whether it also got wrong. If an extractor got faster, count the valid records it produced, not the braces that happened to parse.

Weigh the operational cost too: startup time, upgrade path, observability, rollback, your ability to rebuild the box after a disk dies. A five percent gain that depends on an undocumented patch, or someone manually warming the cache every morning, is a bad trade for a shared box a whole team hits at once, the kind of setup in h100-vllm-team-serving rather than a laptop running one script. Boring configurations keep running next year.

The rule I actually write down, once I've found one, reads something like: schedule long prompts without blowing the interactive latency target. Put the workload, the date, and the reasoning in the result file, and write the condition that should trigger a retest alongside it: a new model family, a driver update, longer contexts, another user, a different traffic mix. Skip that and the benchmark you're proud of today is folklore by next quarter.

Leave headroom past the target instead of chasing full utilization. Spare memory absorbs a prompt twice as long as your test set, and spare queue capacity keeps a batch job from ruining someone's chat session. Thermal and power margin keeps the box from throttling itself into the exact spike you were trying to avoid. A card pinned at one hundred percent isn't efficient, it's one weird input away from an incident.

What I haven't found a clean answer for is telling, from the outside, whether a runtime is actually interleaving prefill and decode or just queuing them under a nicer name. A continuous-batching setup like the one in vllm-at-home and a simpler single-stream server can both report healthy throughput while behaving completely differently once a long document and a live chat land on the same GPU at once. Short of instrumenting the scheduler, you're mostly inferring the policy from its symptoms. I'd like a runtime that just told me, plainly, whose turn it thinks it is. None of the ones I've run do.