Speculative decoding: free speed with strings attached

Speculative decoding is the rare local speedup that doesn't touch output quality: on my 3090 it makes code generation somewhere between 1.6× and 2.2× faster, and the answers are the same answers. I've run it since mid-April. I've also switched it off for half my work, which is the part the enthusiastic posts tend to skip.

the trick in one paragraph

A small draft model runs ahead and proposes a handful of tokens. The big target model then checks the whole batch in a single forward pass: one pass judging several tokens instead of producing one. Wherever the draft guessed right, you pocket those tokens at a fraction of the cost; wherever it guessed wrong, you keep the good prefix, take the target's own choice, and carry on. The verification step preserves the target's output distribution, so quality genuinely doesn't move. The price gets paid elsewhere, in VRAM and configuration patience, and in how well your workload suits the trick.

what it looks like on a 3090

llama-server -m qwen3-14b-q5_k_m.gguf -md qwen3-0.6b-q8_0.gguf --draft-max 8 -ngl 99 -ngld 99

A 14B target with a 0.6B draft from the same model family. On TypeScript refactors and test generation, that pairing runs somewhere between 1.6× and 2.2× faster wall-clock for me, and the server logs show why: draft acceptance hovers around 70% on low-temperature code work.

When acceptance is that high, the big model is mostly nodding along.

the strings

First string: you're loading two models and two KV caches. The 0.6B looked free on paper and cost about 2 GB once its cache and buffers settled in, and that came straight out of my context budget. The 32k I usually run had to shrink to roughly 24k. The VRAM juggling piece is the long version of that pain.

Second string: the pair must share a tokenizer, same family and same vocabulary, or the verification has nothing to stand on. The flagrantly wrong pairings refuse to start, in my experience. The subtly wrong ones just perform badly, which is worse, because nothing tells you.

Third string, the one that got me: draft-size intuition runs backwards. I reasoned that a bigger draft guesses better, so I paired a 3B with my 14B. Acceptance did climb, from about 68% into the high seventies. But the 3B was slow enough on its own that the whole pipeline netted out around 1.2×, and it ate 3 more GB. I ran that configuration for the better part of two weeks before I measured end-to-end instead of admiring the acceptance rate. The 0.6B, accepting less, was faster where it counts.

The draft model's job is to be cheap, not to be smart.

where it collapsed

Code at temperature 0.1 is full of near-inevitable tokens: the import block, the closing brace, the identifier you declared four lines up. The draft nails those, and the speedup is real. Then I left speculation on while drafting prose at temperature 0.8, and acceptance fell to roughly a third; with the draft's overhead, the net came out around 0.9×, slower than running the target alone. Sampling at heat spreads probability across many defensible words, and the draft can't guess which one the dice will pick. So it's on for my code profiles and off for writing now. The toggle is one line, and I've stopped philosophizing about it.

when to bother

Bother when your work is low-temperature and structured, when there's 2 to 3 GB of VRAM honestly spare after the target and your real context, and when you'll measure before and after at your own prompt lengths rather than borrowing mine. Skip it for hot-sampled chat on a maxed-out card. The wider menu of local speedups lives in inference optimization; this one just happens to be the only item on it that quality never pays for.