Capacity-plan a shared local LLM service

Requests per minute is not a capacity number for a shared local LLM box, it just looks like one because it's easy to graph. I still watch it, but I don't let it near a sizing decision.

A two-line chat reply and a forty-page batch extraction count as one request each. They cost the machine nothing alike: how many land at once, how long the output runs, how much context it drags along, whether the right model is already resident in memory. Those move the real number, not the raw count.

same box, three unrelated jobs

Picture a team server carrying a chat interface used occasionally, a coding loop running all day, and a nightly batch job. Treat those as one workload and you size for none of them well.

Before touching a config, capture a baseline: one named model, one fixed prompt set, the exact server command, the model artifact, and prompt template. Those two details vanish first, and a comparison without them means nothing later.

Build workload classes from there, estimate service time per class, cap them, and reserve capacity for interactive traffic ahead of batch. Time to first token covers loading and prompt handling, steady decode rate covers generation, completion time is what a person feels. Add peak memory, queue delay, wall power when they matter. Batch counts finished valid jobs per hour; chat counts waits long enough to complain about.

I keep a log per run: artifact, runtime, launch flags, workload and fixed input, cold/warm start with p50/p95, peak memory and energy, quality failures, a decision with an owner and a retest date. A benchmark without a decision is trivia. A setting without an owner is folklore.

the times-sixty trap

The tempting shortcut: one benchmark multiplied by sixty. It survives review because the output still reads like real text; nobody checks the arithmetic. Local inference fails in soft ways: a model partly falls back to CPU without saying so (check the offload math first), a cache misses, swap creeps upward, a queue keeps serving disconnected clients, a fallback path moves work across a privacy boundary you thought closed. None of that throws an error.

Watch logs and OS counters live, not a summary after. Change one variable at a time unless comparing full configurations. Run it more than once; one fast pass proves nothing. Read what the model wrote instead of trusting latency alone: a change that buys speed at quality's cost gets logged too.

Simplicity belongs in the same ledger. A clever setting needing manual repair after every driver bump isn't free. I'd rather run the boring version that comes out of a service file twice, unchanged. Drop raw artifacts, keep hashes and commands, so a rerun months later still means the same thing.

spend headroom on purpose

The rule that holds: plan off measured distributions and put admission control in front of the server, rather than reach for the biggest model or the newest accelerator. It's duller, but the stack's limits stay visible. A visible limit gets routed around, scheduled, or priced. An invisible one shows up as a stuck queue on an ordinary Tuesday.

It's the same case a team running vLLM across a shared H100 pair makes putting a queue in front of the model instead of trusting every caller.

The tradeoff I take on purpose is idle capacity. I size the box to stop well short of its measured ceiling, and most of that spare room sits unused on an average afternoon. That's the price: I give up squeezing every last token out of the hardware because the alternative is finding the ceiling mid-demo, with no margin left.