llama.cpp gets DeepSeek V4's Lightning Indexer with an f16 Metal path

On August 3 two commits landed in llama.cpp that matter more for Apple Silicon than anything in the July release notes: support for DeepSeek V4's Lightning Indexer, and an f16 variant of that indexer for the Metal backend. Two days earlier, on August 1, a --reasoning-preserve flag arrived that keeps reasoning content in the conversation history, and on August 2 the WebGPU backend got f16 support. By August 7 the rolling build was b10322 and the sparse-attention path was still being iterated.

What the Lightning Indexer does

Full attention costs O(n²) in the sequence length. At 1M tokens that is not a rounding error; it is the reason 1M context on a laptop has mostly been a spec-sheet number. DeepSeek V4's answer is an indexer: a lightweight component that scores tokens and selects the relevant subset for each query, so the expensive attention runs over a fraction of the context rather than all of it. The model keeps its roughly 1M window, and the runtime pays for the tokens the indexer picks.

llama.cpp now implements that indexer, and the f16 build gives Metal a native path instead of falling back to a slower or higher-precision kernel.

Why f16 on Metal is the interesting bit

On an M-series machine there is no separate VRAM. Weights, KV cache, indexer state and your IDE all share one pool, and the GPU's decode speed is bounded by how fast it can stream through that pool. Two things follow.

• Memory. The indexer's own state is small compared to the KV cache, but running it in f16 halves what it would cost in f32 and, more importantly, keeps it in a format the Metal kernels are fast at. The KV cache is still the big number; the local KV cache math does not change, but the indexer changes how much of that cache attention has to read per step.
• Latency. Sparse selection turns attention from a full pass over the cache into a pass over the selected tokens. In practice that should show up as a much flatter decode curve as context grows, which is exactly where llama.cpp on Metal has been losing to shorter-context workloads.

Sparse attention does not make 1M context free on a Mac. It makes the cost proportional to what the model looks at, which is the first time that number has been under your control.

The trade-off is recall. An indexer that selects the wrong tokens produces a model that is fast and wrong about something 400K tokens back. That is measurable, and it is the thing to measure first.

What to measure before you switch

I would treat the August 3 build as a benchmark target, not a default.

• Decode tokens per second at 8K, 64K, 256K and as far as your RAM allows, dense versus indexer, same quant, same prompt, median of three runs with a cooldown.
• Peak resident memory at each point. The KV cache will dominate; confirm the indexer is not adding more than you expect.
• A needle-style recall test at each context length: plant facts early, ask late, score. This is the acceptance rate of the indexer in disguise.
• Prefill time separately from decode. Sparse attention helps decode; prefill still has to build the cache and run the indexer over everything.
• --reasoning-preserve on and off for agent loops. Keeping reasoning in history improves continuity and inflates context, so it interacts directly with the indexer's selection.

For the model side, DeepSeek V4 already runs with 1M context by default via the API; the point of the local path is data that cannot leave the machine, not beating the API on speed. If you are choosing a runtime, MLX versus Metal still comes down to whether you need MLX's throughput or llama.cpp's control.

The honest gap

I have not published numbers here because I do not have them yet: the commits are days old, the build is rolling, and the release material frames recall versus footprint at long context as an open question rather than a resolved one. Expect the first community benchmarks to be noisy and expect the indexer to behave differently across quantizations. What is not in doubt is the direction: sparse attention with a native Metal path is how 1M context stops being a marketing number on Apple Silicon.