M5 Neural Accelerators: 4x time-to-first-token, and why decode barely moves

The M5 numbers Apple's ML research group and independent testers published this week split cleanly into two columns, and most coverage only prints the first one. Prefill: through MLX, Qwen3-14B at 4-bit gets time-to-first-token 4.06x faster than on M4. Decode: the same model generates tokens 1.19x faster. One is a step change, the other is a bump. If you run models on Apple Silicon, which column your workload lives in decides whether M5 changes anything for you.

What is actually new in the silicon

The M5 GPU has a Neural Accelerator inside each of its 40 cores, part of what Apple calls the Fusion Architecture, and MLX reaches it through Metal 4 TensorOps. That is where the prefill gain comes from: prompt processing is a dense matrix workload, and the accelerators do dense matrix work. Around it Apple lists 153 GB/s of memory bandwidth on M5 Max, up 28% on M4, an AMX 2.0 block, and a 45% improvement in FP16 efficiency. The llama.cpp Metal backend does not currently reach the accelerators, which is why the MLX versus llama.cpp gap widened on M5 specifically: the follow-up measurements put MLX 30-60% faster overall on M5 with 3-4x prompt processing.

Why decode barely moves

Token generation is memory-bound. Every decode step streams the active weights through the memory system once, so tokens per second is bandwidth divided by bytes per token, give or take. A 28% bandwidth bump therefore buys you something in the region of 20-30% on decode, and 1.19x is right where you would expect. Above roughly 27B parameters the steady-state decode numbers for MLX and llama.cpp converge on M5, because at that size neither runtime is compute-limited and both are waiting on the same 153 GB/s. Extra compute in the GPU cores cannot help a phase that is not compute-bound.

Prefill is a compute problem and Apple just bought compute. Decode is a bandwidth problem and bandwidth went up 28%.

The concrete reference points from the July measurements: Gemma 4 E2B runs at about 158 tokens/s on M5 Max under MLX, and GLM-5.2 Air runs at around 30 tokens/s on a 64 GB Mac. Ollama separately reported Gemma 4 generation nearly 90% faster on Apple Silicon, but that comes from multi-token prediction and automatic tuning in its refreshed MLX and llama.cpp engines, not from the M5 hardware.

Which workloads benefit

• RAG and long-prompt agents: big win. A 20K-token prompt that used to take seconds to prefill now takes a quarter of that; the answer arrives at nearly the same tokens per second after.
• Short-prompt chat: small win. TTFT was already fine, and the 1.19x on decode is what you feel.
• Batch summarisation of long documents: big win, since it is almost pure prefill.
• Code completion with large repo context: big win on the first token, ordinary on the rest.
• Models over ~27B: treat M5 as an M4 with slightly more bandwidth; the runtime choice matters less than the memory you bought, which is the unified memory sizing question again.

One caveat from the earlier MLX versus llama.cpp studies still applies: the MLX advantage shrinks past roughly 40K tokens of context, where llama.cpp holds up better. If your prompts are that long, measure both.

What to measure on your own machine

Split every benchmark into TTFT and tokens per second and report them separately. Run at least one model under 14B, one around 27B, and one bigger, so you see where the curves converge. Log which backend actually executed the matmuls; on M5 that is the difference between the accelerators being used or not. Keep prompt lengths fixed across runs, since prefill scales with prompt length and decode does not. The Neural Engine versus GPU question is a separate axis: the accelerators here live in the GPU cores, not in the ANE.

The honest gap

The 4.06x figure is Apple's own measurement on one model at one quantisation, and the 30-60% and 3-4x follow-ups are from a single July benchmark set. Nobody has yet published a full sweep across model sizes on M5 Max showing exactly where the prefill advantage decays, and the 27B convergence point is a reported observation, not a curve. Treat the direction as settled and the exact multipliers as something to reproduce before you put them in a purchase justification.