Ollama's automatic MTP speculative decoding on Apple GPUs: what the 90% actually covers

Ollama's August build does something I have wanted from a local runtime for a while: it switches on speculative decoding for Qwen3.5 on Apple GPUs automatically, using the model's own multi-token prediction head as the draft. No second model to download, no flag to remember. It sits on top of the July work, where Ollama reported Gemma 4 running almost 90% faster on Apple Silicon through multi-token prediction and automatic tuning on refreshed MLX and llama.cpp engines. Those are big numbers. They are also numbers about one half of your workload, and that is the part worth understanding before you plan around them.

What shipped in August

• MTP-head speculative decoding for Qwen3.5 on Apple GPU, enabled automatically in the MLX engine.
• The draft output head is quantized to the same type you requested for the model, so the draft path does not silently run at a different precision.
• A fix for Qwen3 MoE decoding when experts are quantized at different levels, which had been misbehaving in mixed-quant builds.
• Roughly 4-9% faster packed gate/up projection on M5 Max, per Ollama's release notes.

How an MTP head replaces a draft model

Classic speculative decoding pairs a small draft model with your target. The draft proposes a few tokens, the target verifies them in one forward pass, and you keep the prefix that matches. It works, but you pay for a second model in memory, and on a Mac with unified memory that is real capacity you are giving up. A multi-token prediction head is a cheap extra output layer trained to guess the next few tokens from the same hidden state. Used as the draft, it costs almost nothing in memory and shares the target's representation, so its guesses tend to align better than an unrelated small model's. The catch is that its quality is fixed at training time; you cannot swap in a stronger drafter when acceptance drops.

Speculative decoding does not make the model faster; it lets the model verify more tokens per step, and the win is exactly as large as the acceptance rate allows.

Acceptance rate is the whole story. On predictable text, code, boilerplate, structured output, the head guesses well and you get most of the theoretical multiplier. On high-entropy output the target rejects more drafts and the drafting overhead eats the gain. I walked through the arithmetic in speculative decoding break-even; the August build changes the constants, not the shape of the curve.

Prefill is a different workload

The 90% and the 4-9% both describe decode. Prefill, the pass that ingests your prompt before the first token appears, does not benefit from speculation at all, because there is nothing to speculate about. On M5 the prefill story is separate: Apple's own measurements for Qwen3-14B at 4-bit show time to first token 4.06x faster than M4 and generation only 1.19x faster, thanks to the Neural Accelerators in each GPU core. So the hardware favours prompt-heavy work and the software favours generation-heavy work, and which one you feel depends on what you run. A RAG pipeline stuffing 30k tokens of context in front of a two-line answer is prefill-bound and sees the M5 gain, not the MTP gain. A coding agent emitting long diffs is decode-bound and sees the opposite.

What to measure on your own Mac

• Time to first token and tokens per second separately, on the same prompt, with and without speculation. Never a single blended number.
• Acceptance rate per workload class: code, JSON, prose.
• Peak memory with the MTP path active versus any two-model setup you configured by hand.
• Output equality. Speculative decoding should be lossless with greedy sampling; a diff between runs means something is wrong in the quant path.
• Sustained throughput over ten minutes, not a five-second burst, because thermal behaviour on a laptop changes the answer.

The honest limit

Every figure above is vendor-run: Ollama's release notes for the 4-9% and the 90%, Apple's research post for the M5 prefill numbers. Nobody has published acceptance rates for the Qwen3.5 MTP head across workload types, and the automatic enablement covers Qwen3.5 only, so Gemma or Llama through the same engine are still on the older path. Measure your own prompts before you tell anyone it got 90% faster.