SpecPilot: an agent doubled speculative decoding speed, and the 10.87x needs context

The SpecPilot headline says speculative decoding goes from 4.46x to 10.87x over plain autoregressive decoding, without training and without quality loss. I distrust any decoding number above 10x, so I went to the repository instead of the abstract. The number holds up. It is also the best of six settings, measured on a single Blackwell workstation card, and the more interesting result is the method that found it.

What SpecPilot is

The paper, arXiv 2611.02501 by Li, Chai, Liang, Hu, Xu, Pan, Zheng, Ramchandran, Keutzer and Jiao, opens with a claim I agree with: released speculative decoding systems leave much of their speed unused, and the best fix differs for every deployment. SpecPilot is an agentic loop that searches for lossless optimizations per deployment, meaning per target model, drafter, GPU and serving mode.

The README describes the loop. In each round a cost model locates the largest gaps between the current system and its theoretical ceiling. Parallel agents explore directions and integrate the ones that work, with a kernel sub-agent for kernel tuning. Token forcing checks every output token against the stock system to catch drift from the target model. A structured memory carries validated mechanisms into later rounds and other deployments. Every shipped engine decodes greedily and was accepted only after a token-level check that the target model chose each committed token.

That token check is the design decision I would copy. An agent told to maximize tokens per second will find a way to emit fast garbage if nothing stops it.

Where the numbers come from

The repository ships the engines SpecPilot found for six deployments, on a pruned SGLang tree:

All values are speedups over autoregressive decoding, single request unless marked batched, where the batch sizes were 16, 20 and 24. Over stock the gain is 1.93x to 2.52x in every row, and that is the honest headline: tuned per deployment, the same drafters roughly double their speedup. The 10.87x is the best single-request row, on the RTX PRO 6000 Blackwell. On H100 the single-request results are 8.86x and 9.41x. A fresh-clone reproduction on four H100 settings landed between 1.94x and 2.52x over stock.

The mechanisms it found

None of them is exotic, which is the point. W12 is a lossless 12-bit weight format, about 12.01 to 12.02 bits per weight on dense models, decoded inside the GEMM registers. In a memory-bound decode, reading 12 bits instead of 16 cuts weight traffic by a quarter. Two-level verification uses an 8-bit or INT6/INT8 copy of the weights. Then tree verification, a certified exact-argmax LM head and L2 weight prefetch. For batching it found ragged draft trees, a tiered verification budget, FP8 GEMM routing, segment admission for prompts and a per-request drafter block length.

Each one is the kind of change an experienced inference engineer might find in a week of profiling on one deployment. The agent's value is doing it for every deployment.

The finding is less the 10.87x than the fact that stock speculative decoding leaves about half its speed unused, and the missing half is specific to your hardware.

The caveats the README states

Outputs are not bitwise identical to autoregressive decoding for any speculative decoder in the repo, stock included, because batched kernels reduce in a different order and near-ties flip. Benchmark scores are described as a sanity check rather than an accuracy evaluation, and many answers hit the 2,048-token cap. Kernels are hardware-specific: the Blackwell engines are built for sm_120a only, and the H100 engines were measured only on H100. The stack needs torch 2.13 and CUDA 13.0, and the first run compiles kernels and is 10 to 15 percent slower, so warm up before you measure.

What the README does not tell me is the price of the search: no GPU hours, no agent model named, no instructions for running the loop on a new deployment. You get six finished engines, not the factory. My speculative decoding break-even notes still apply, because a 2x speedup that costs a week of agent and GPU time per deployment pays back very differently from one found in an afternoon.

If you serve Qwen3.8-27B on H100 or RTX PRO 6000 with DFlash2 or DSpark, clone the repo, run its bench and compare scripts against your stock setup, and measure on your own prompts. If you serve anything else, take the method: profile against a cost-model ceiling, gate every change on token-level equivalence, and expect your tuned optimum to differ from the library default.