Inference optimization: how local model serving gets fast

You've got a local model running. Whether it's usable comes down to tokens per second, and that number depends on the serving stack at least as much as the hardware. Two setups with the identical model and GPU can differ several-fold. Here's what matters.

The one fact everything follows from

Token generation is memory-bandwidth-bound, not compute-bound. To produce each new token, the model reads its entire set of weights from memory. So your tokens-per-second is gated by how fast you can move weights, not by raw FLOPs. Two consequences fall straight out:

• Quantization is the cheapest speedup. int4 weights are a quarter the bytes of fp16, a quarter the memory to move per token. That makes it faster, not just smaller. (And it fits, which is the other half.)
• Memory bandwidth predicts the experience. It's why an Apple Ultra chip or a high-bandwidth GPU feels fast beyond its FLOPs (hardware guide).

The KV cache: the hidden memory cost

Attention needs the keys and values of every previous token. Recomputing them each step would be quadratic, so they're cached: the KV cache. Essential, but it grows with context length and lives in the same precious memory as the weights. The practical upshot most people miss:

Past a long-enough context, the KV cache can rival or exceed the weights in memory. "Why did my long-context request OOM when the model fit fine?" is almost always the KV cache. Long context isn't free even when the weights are small.

The serving-tier techniques

When you go from one user to many (the vLLM world), three techniques do the heavy lifting:

• Continuous batching. Instead of processing requests one at a time, interleave many so the GPU stays saturated. This is the key to throughput (serving a team), and it's why a serving engine beats running the model in a loop.
• Paged attention. Manage the KV cache like virtual memory, in pages, so you don't waste memory reserving worst-case contiguous blocks. It lets you fit more concurrent requests in the same VRAM.
• Speculative decoding. A small, fast draft model proposes several tokens; the big model verifies them in a single pass. When the draft guesses right (often), you get multiple tokens per expensive step: a single-stream latency win, not just throughput.

What to actually reach for

Match the stack to your situation:

• One developer, want snappy responses → quantization (int4) + good-bandwidth hardware, via Ollama / llama.cpp / MLX. These already do KV caching and reasonable defaults; speculative decoding where supported is a bonus.
• Serving a team → a serving engine (vLLM) with continuous batching and paged attention. You're optimizing throughput and concurrency, a different problem from single-stream speed.
• Long-context workloads → budget for the KV cache, not just the weights; it's the memory that surprises you.

The lazy take

For one person, just run Ollama with an int4 model. It handles caching, batching, and sane defaults, and it'll be fast enough on hardware that fits the model. Don't stand up a vLLM cluster and tune paged attention for a single-user setup; that's serving-tier machinery for a problem you don't have. Reach for the heavy stack when you're serving many users and tokens-per-second or concurrency becomes the actual bottleneck, and let a profiler, not a blog post, tell you when that is (ponytail). The fundamentals are bandwidth-bound generation, quantization, and the KV cache — the rest is tuning you add when the load demands it.