vLLM at home: throughput machine in a latency world
The ticket pipeline from last week's post classifies tickets as they arrive, one at a time, and Ollama is exactly right for that shape of work. Then the client asked for the archive: ten weeks of history reclassified under the new taxonomy, 10,212 items, re-run nightly so the dashboards stay comparable while the taxonomy keeps shifting. Wasteful? A little. Their call. My Ollama setup with four parallel workers ground through the first chunk at a measured 31 items a minute, which projects to five and a half hours. I killed it at the 90-minute mark and finally installed vLLM.
What vLLM is actually built for
vLLM's two headline tricks matter more at home than the papers made me expect. PagedAttention stores the KV cache in fixed-size pages instead of one contiguous slab per request, so hundreds of in-flight sequences can share a 24 GB card without fragmenting it to death. Continuous batching lets requests join and leave the running batch at every decoding step, so the GPU never drains and refills between jobs. Neither trick makes a single answer arrive sooner. Together they make the card produce dozens of answers at once at close to the speed of one, and a good chunk of inference optimization is variations on that theme.
Numbers from the nightly run
Same 3090, same 8B model, though not the same file (more on that in a moment). Each item runs about 700 tokens in and 40 to 60 out. Ollama with four workers: 31 items a minute, five and a half hours projected. vLLM serving the same model as AWQ with 256 requests in flight: the full 10,212 in 47 minutes, GPU pinned between 96 and 98% the whole run. An even 7× on wall clock, which I distrusted enough to rerun the next night. Second run: 49 minutes.
Single-request latency, meanwhile, got worse.
The friction the quickstart skips
vLLM doesn't do GGUF in any way I'd rely on, so my tidy Q4_K_M collection was useless here; I pulled an AWQ build of the same model instead, another 5.6 GB of disk. The install ate an evening on the torch-and-CUDA wheel dance before I stopped improvising and let uv resolve the whole thing. Cold start runs 70 to 90 seconds on my box, CUDA graph capture included, which feels geological if Ollama's lazy loading is your baseline. And the default gpu-memory-utilization setting of 0.9 grabbed about 21.5 GB the instant the server came up, which OOMed immediately because the desktop session already held 1.4 GB for the browser and compositor. Mine lives at 0.82 now.
The week I used it as a desk lamp
Flush with the backfill win, I spent a week trying to make vLLM my only server: one engine, one port, a tidier life. The conviction lasted five days. While vLLM is up, the card is spoken for. That preallocated pool means Ollama can't load so much as an embedding model beside it. Swapping models is a restart, not a request parameter. And for one person typing questions, it bought me nothing over Ollama except a slower cold start. A throughput machine spent 97% of its day idle with 21 GB reserved — a bus I'd bought to commute in alone.
Ollama answers a person; vLLM answers a queue. Most of my week is a person, and almost exactly one hour of it is a queue.
So now it's a systemd timer. Ollama owns the card all day; at 02:10 a unit stops it, brings up vLLM, runs the backfill and a small eval sweep, and hands the card back before three. Same shape as my overnight agent runs: local hardware doing queue work while nobody stands in the queue.
Two warnings to close. vLLM's docs and defaults assume a data center, so consumer-card answers mostly live in GitHub issues: budget a real evening for setup, two if your CUDA install has history. And don't take my 7× as yours. Batch gains swing hard with input and output lengths, and measuring your own tokens is the only benchmark that has ever transferred between two people's machines. For its one hour a night in my basement, though, nothing else comes close.