From the RTX lab to the H100s: what transfers and what doesn't

We run a two-environment shop: two Blackwell RTX PRO 6000 test boxes where models audition, and the H100 NVL pair in the DL380 where models work. The reasons for that split are economic; this post is about the seam between the two: the promotion pipeline, and the hard-won list of what survives the trip from sm_120 to sm_90 and what silently does not.

I keep this list on an actual wiki page at work, because every item on the "does not transfer" half cost someone at least one afternoon.

what transfers cleanly

More than you might fear, which is what makes the cheap test bench viable at all.

• Model quality verdicts. This is the big one, and the reason the whole setup works. "This model is good at our extraction tasks and mediocre at code review" is a property of the weights, not the silicon. A model that passes our eval suite on a test box passes it on the H100s, same precision assumed. In eighteen-ish months we have never seen a quality verdict flip between environments at matched quantization.
• Chat template and tokenizer quirks. The model that needs a system prompt in a weird slot, the tokenizer that splits diff markers annoyingly, the template that silently eats trailing whitespace: all of it turns up on test and reproduces exactly on prod. Debugging these on the cheap boxes instead of production is pure profit.
• Prompt and eval suites. The suite itself is environment-agnostic by construction. We run the identical harness against both, which is also our end-to-end check that a promotion actually worked.
• The weights. Safetensors checkpoints are portable; we pull once to the NAS and both environments load the same files. Boring, as it should be.
• vLLM config in spirit. The shape of the config (chunked prefill on, roughly this scheduler pressure, this max-model-len philosophy) carries over as a starting point. The numbers inside it do not, see below.

what does not transfer

• Performance numbers. The test cards push ~1.8 TB/s of GDDR7; the H100 NVL pushes ~3.9 TB/s of HBM3, and the two saturate completely differently under batching. We measured single-stream ratios around 2.1x and batched-throughput ratios that were nothing like that. Rule on the whiteboard: test-bench tok/s may not appear in any capacity plan. Ever. People tried. It went badly.
• Tensor-parallel configs. Prod serves big models at TP=2 over NVLink. The test bench has no NVLink, so everything there runs single-card. That means TP=2-specific behavior (the memory split, the latency profile, one genuinely cursed non-divisible-heads edge case) is only ever discovered on production hardware. We schedule that discovery deliberately, off-hours, instead of letting it happen to us.
• The quantization format itself. Test-bench NVFP4 experiments from the quantization lab cannot be served on Hopper, which has no FP4 hardware. Promotion to prod means re-quantizing to FP8: a different format, a different artifact, and quality that can shift in either direction. So models get their gate evals run in FP8, the format prod will actually serve, regardless of what else we measured along the way.
• TensorRT-LLM engines. Compiled per-architecture. An engine built on sm_120 will not load on sm_90. Not degraded, not slow. Will not load. Obvious once you know; cost a colleague half a day of confusing errors before we knew. Engines are build artifacts of a specific GPU, full stop. This is also quietly why vLLM, which loads portable checkpoints, is our default and compiled engines are the exception.
• CUDA and driver assumptions. The environments run different driver trains more often than not, because the test bench absorbs upgrades first by design. "Works on test" therefore includes a version skew you must consciously subtract before concluding anything about prod.

Correctness and quality are properties of the model. Speed is a property of the machine. Promote the first, re-measure the second.

the promotion checklist

A model becomes a production option by clearing four gates, in order, and the order exists because each gate is cheaper than the next.

• Gate one: task evals on the test bench, in FP8. Full suite, scores within our thresholds against the incumbent. Most candidates die here, which is exactly where dying is cheapest.
• Gate two: license cleared. Someone actually reads the license against our use, per the process in local model licenses. Fifteen minutes of reading, done before we get attached. We have dropped one otherwise-excellent model at this gate and I remain a little bitter about it.
• Gate three: fits the prod memory budget, and that means more than fitting on the card. It has to fit alongside the KV cache headroom our real concurrency needs, inside the budget the H100 pair actually has, with everything else it serves. A model that fits but starves the KV cache is a model that does not fit.
• Gate four: a canary weekend. The model goes live on prod serving Friday evening as a selectable option, per the team-serving setup, and volunteers use it for real work while the incumbent stays default. Monday we read logs, latency percentiles, and complaints. Only after a clean weekend does it become a default anything.

the one that fooled us

The war story the checklist grew around. A coding model went through triage last winter and looked genuinely great: best scores we had seen on our editing evals, everyone excited. On the test bench we had been running our NVFP4 artifact of it, because that is what the lab produces, and single-stream it felt superb.

Promotion meant re-quantizing to FP8 for Hopper. The FP8 artifact scored about 1.2 points lower on our hardest code eval than the NVFP4 one had, on our suite, on our hardware. Not catastrophic, but our margin over the incumbent had been about a point, so the entire advantage evaporated in requantization. Different format, different rounding, different model, effectively. That incident is the reason gate one now mandates FP8: evaluate in the format you will serve.

Same model, second lesson: under the canary weekend's real concurrency, mean latency looked fine while p95 quietly doubled versus the incumbent. Single-user feel on a test box says nothing about scheduler behavior under eight simultaneous users on different silicon. The canary caught it; we retuned max-num-batched-tokens on prod and it settled. But "felt fast on my box" is now a banned phrase in promotion discussions.

the limitation: discipline costs, and we sometimes decline to pay

Two-environment discipline is real overhead, and a five-person team feels every gram of it. The full pipeline takes two to three elapsed days even when every gate passes first try. When a model drop is exciting and someone wants it today, the pressure to skip straight to prod is enormous. I will not pretend we have never yielded. We skipped the canary once for an "obviously fine" minor version bump of a model already in production. It shipped a subtly different chat template; tool-calling broke for two people for most of a day before anyone connected the dots. Small blast radius, maximally instructive.

So the honest close is this: most promotions transfer cleanly, and for that ninety percent the checklist can feel like ceremony. It exists for the tenth one, and you cannot tell which one is the tenth from the outside. That is the entire problem it solves. Write down your own transfers-list, keep the gates cheap enough that skipping them stops being tempting, and when you do skip one anyway, at least write down what it cost. Ours says two people, one day, tool-calling. The checklist has been cheaper ever since.