NVFP4 on the Blackwell test boxes: quantization as a pipeline, not an event

There is a mildly absurd fact at the center of our fleet: the cheap test cards can do something the expensive production cards cannot. The RTX PRO 6000 Blackwell has native FP4 tensor cores. The H100 NVL, for all its HBM3 glory, does not. Hopper simply has no FP4 hardware. So NVFP4, the format everyone is converging on for 4-bit inference, runs natively on our Blackwell test bench and nowhere in production.

We decided to lean into that instead of ignoring it. The test boxes became the quantization lab, and this post is about the pipeline that grew there, with one honest caveat up front: every NVFP4 win we produce is, today, a test-bench-only win.

what NVFP4 actually is, in one paragraph

NVFP4 is 4-bit floating point with microscaling: E2M1 values in small blocks, each block carrying an FP8 scale, plus a coarser per-tensor scale on top. The two-level scaling is the whole trick, letting a format with only sixteen representable values per element track the local dynamic range of real weight tensors far better than the older block formats did. On Blackwell, the tensor cores consume this directly, so you get the memory savings and a compute path, not just a storage trick that dequantizes to something else before every matmul.

The practical consequence: weights at ~4.5 bits effective, roughly half the footprint of FP8, on hardware that treats it as a first-class citizen.

why "pipeline, not event"

The old workflow (everywhere, including here, for a while) treated quantization as an event. Someone converts a model once, spot-checks a few prompts, uploads the artifact, and it lives forever. Six months later nobody remembers which calibration set was used, the tooling that produced it has had nine releases, and the artifact is a small archaeological mystery.

We now treat it like a build. There is a repo. In it: the recipe per model (quantization scheme, calibration dataset, sample count, which layers are excluded), the tooling versions pinned, and the eval results checked in next to the config that produced them. When the tooling updates or the model gets a new checkpoint, we re-run the pipeline and diff the evals. The artifact is disposable; the recipe is the asset.

The concrete loop, for a model we serve in FP8 on the H100s:

• Pull the original weights and our FP8 production config.
• Produce an NVFP4 checkpoint with llm-compressor (the TensorRT Model Optimizer path works too; we've used both, and which one behaves better changes month to month, which tells you something).
• Load it in vLLM on a test box and run the full eval suite against the FP8 baseline running on the same box.
• Check the numbers into the repo. Only if the deltas are acceptable does the artifact become a candidate for anything real.

Same box for both sides of the comparison, deliberately. Comparing NVFP4-on-test against FP8-on-prod would smuggle in every hardware and kernel difference at once.

perplexity is not enough

Every quantization README in existence reports perplexity, and perplexity is nearly useless for the decision we actually face. We have seen an NVFP4 conversion land within about 1.5% of baseline perplexity while dropping roughly 4 points on our internal code-editing eval, on our models, on our tasks. The model still spoke fluent English; it had just gotten measurably worse at the one thing we pay it to do.

So the gate is task evals: the code tasks, the extraction tasks, the structured-output checks, the multilingual bits that matter to us, the same suite we described in measuring local tokens. Perplexity is recorded as a smoke test, nothing more. If perplexity explodes, the conversion is broken. If perplexity is fine, you know nothing yet.

A quantized model that keeps its perplexity and loses its competence is the default failure mode, not the rare one.

And the degradation is wildly model-dependent, which is why there is no blanket policy. Dense models in the 30–120B range have mostly taken NVFP4 gracefully for us: deltas within noise on maybe two-thirds of our evals, a point-ish drop on the hardest reasoning tasks. MoE models are the wild cards. Architectures with many small experts concentrate a lot of behavior into individually tiny weight matrices, and 4-bit noise in a small expert is proportionally louder; we have one MoE where NVFP4 was essentially free and another where a specific tool-calling eval fell off a cliff while everything else held. Same format, same recipe, opposite verdicts. Per-model evals or nothing. Anyone selling you "FP4 is fine now" as a general claim is averaging over models they do not serve.

what 4-bit buys on a 96 GB card

The selfish, immediate payoff is that the test bench can hold models it otherwise could not.

Rough arithmetic, hedged as always: a 235B-class MoE (Qwen3-235B territory) comes out somewhere around 120 GB of weights in NVFP4. Still does not fit a single 96 GB card, and no, two test cards over PCIe is not an answer for interactive use since there is no NVLink on the bench. But a 120B-class dense model, or a GLM-Air-class MoE, lands in the 55–65 GB range in NVFP4. That fits on one card with genuinely comfortable room for KV cache and long contexts. Models we could previously only evaluate on the production pair (carefully, off-hours, per what fits on the H100 pair) are now a casual test-bench load.

That alone justified the pipeline. Triage of big models stopped competing with production for slots.

the honest part: production cannot use any of this

The H100s cannot run NVFP4 natively. Hopper has no FP4 tensor cores, full stop. There are emulation paths that dequantize on the fly, but you keep the quality risk and lose the performance point, so we do not bother. Production serves FP8, which on Hopper is mature, fast, and boring in the best way.

So the pipeline earns its keep in two ways, today. First, the capacity win above: bigger models on the test cards, now. Second, and I think more importantly: readiness. Our next GPU purchase will be Blackwell-generation or later, whatever form that takes, and on day one of that hardware we will already have a year-plus of accumulated per-model evidence about which of our models survive FP4 and which do not, with recipes we trust because we have re-run them twenty times. The alternative is starting that learning curve the week the new hardware arrives, under pressure, which is exactly when you make blanket assumptions.

It also feeds the promotion process in a sideways way. NVFP4 verdicts do not transfer to prod, and the test-to-prod pipeline post covers precisely which conclusions survive the trip and which get re-derived.

the limitation: the ground moves monthly

The thing that actually bit us is tooling churn. The quantization stack (llm-compressor, Model Optimizer, the vLLM kernels that consume the checkpoints) moves monthly, and recipes rot. A recipe that produced a clean artifact in March emitted a subtly different scale layout by June; the checkpoint loaded fine and scored about 2 points lower on our evals, on our hardware, with zero errors or warnings anywhere. Silent, loadable, worse. We only caught it because re-running evals on every regeneration is mandatory, and that rule exists because of this exact incident.

So: pin your tooling versions in the recipe, re-run the full eval suite on every regeneration even when nothing "should" have changed, and treat any quantized artifact whose evals you cannot reproduce as untrusted. Quantization is not a checkbox you tick once per model. It is a build product, and build products get rebuilt, and rebuilds get tested. Run it like that from the start and the monthly churn becomes a diff to review instead of a mystery to debug.