FP8 delayed scaling in NVIDIA Transformer Engine: when it pays and how to measure BF16 vs FP8

This one is a reference, not news. NVIDIA's Transformer Engine is the standard way to run transformer training and inference in FP8 on Hopper-class and newer GPUs, and a technical guide on configuring its fused kernels, FP8 delayed scaling, and benchmarking BF16 against FP8 was circulating in early August. Here is what delayed scaling is, when it pays, and how to measure it without fooling yourself.

Why FP8 needs a scaling strategy at all

FP8 has eight bits. The two common formats, E4M3 and E5M2, trade mantissa precision for exponent range in opposite directions, and neither has enough range to hold raw activations or gradients as they come off a layer. So every FP8 tensor is stored alongside a scaling factor: the value is scaled into the representable range before the cast and scaled back after. Pick the factor badly and you saturate large values or squash small ones to zero, silently.

The obvious approach is to compute the maximum absolute value of the tensor you are about to cast, the amax, and derive the scale from it. That is current scaling, and it has a cost: a full pass over the tensor to find the amax before you can write it, which serializes the cast behind the reduction.

What delayed scaling does

Delayed scaling breaks the dependency. Instead of the current tensor's amax, it uses a history of amax values from previous iterations. Transformer Engine keeps a rolling window of past observations per tensor, applies a reduction across that window (the maximum is the conservative default), and uses the result to compute the scale for the next cast. The current iteration's amax is recorded afterwards and enters the window for later steps.

The bet is that activation and gradient magnitudes drift slowly during training, so yesterday's range is a good enough predictor of today's. When that holds, the cast becomes a single fused pass with no preceding reduction, and the fused kernels can combine the cast with the matmul it feeds. That fusion is where the throughput comes from; the eight-bit storage is only part of it.

FP8 speed does not come from the smaller number. It comes from what you no longer have to compute before you are allowed to write it.

When it pays

• Large, stable training runs. Once past warm-up, magnitudes settle and the history is a good predictor. This is what the technique was designed for.
• Matmul-dominated layers. The gain is in the GEMMs. Attention softmax, layer norm and embedding lookups stay in higher precision and see no benefit.
• Prefill-heavy inference. Half the bytes per weight matters when you are streaming weights for a long prompt. Small-batch decode is bound by the KV cache, so the win is smaller.

Where it does not pay: early training with spiky gradients, fine-tunes that shift the distribution fast, and any workload where matmuls are a minority of wall time. The wider pipeline picture for Blackwell-class hardware is in the NVFP4 pipeline, and the generational split in Hopper vs Blackwell.

How to measure BF16 vs FP8 honestly

Most published comparisons report one throughput number, mix prefill with decode, and ignore power. My checklist:

• Separate prefill and decode. Fix a prompt length and a generation length, and time them independently. FP8 helps the two phases differently.
• Hold the batch constant. Compare at the same batch size, then sweep. A bigger batch that only fits because FP8 halved memory is a real win, but a different experiment.
• Record tokens per second and tokens per joule. Power at the wall or from the GPU counters. A kernel that is faster but pulls proportionally more power has not improved your bill.
• Check the loss curve, not just the speed. Train BF16 and FP8 on the same seed and data order long enough to see divergence. If the FP8 curve lifts, tune the amax history length or reduction before trusting the throughput.

The honest limitation

I have deliberately given no percentages, because I have none I would sign. The guide that prompted this post is a configuration reference, not a benchmark, and the gains depend on model size, sequence length, batch and the fraction of time spent in GEMMs. Treat any single "FP8 is X times faster" figure as the vendor's best case until you have reproduced it on your own model with prefill and decode measured separately.