Why our test bench is Blackwell RTX, not more H100s

When the budget line for test hardware finally cleared, everyone on the team assumed we would buy a third H100. That was the obvious move: same card as prod, no surprises, one more slot in the DL380. We did not do that. We bought two RTX PRO 6000 Blackwell workstation cards instead, stuck each one in an unremarkable tower chassis, and I want to explain why I think that was the right call. And where it bites us.

The production box is the machine this whole blog orbits: an HPE ProLiant DL380 Gen11 with a pair of H100 NVL cards, NVLink-bridged, roughly 192 GB of HBM3 between them. That machine has a job. It serves models to the team all day and runs batch work overnight. Every hour it spends helping someone debug a driver update or triage some random Tuesday model drop is an hour of production capacity we burned on work that did not need HBM3.

That is the entire argument, honestly. The rest of this post is detail.

the 96 GB parity is the whole point

The RTX PRO 6000 Blackwell has 96 GB of GDDR7. The H100 NVL has 96 GB of HBM3. That number matching is not a coincidence in our purchasing decision: it is the reason the purchase works.

If a model fits on one test card, it fits on one prod card. Same weights, same quantization, same KV cache budget, same --max-model-len before you hit the wall. When someone runs a new 70B-class checkpoint on the test bench and reports "fits in FP8 with about 21 GB left for KV," that sentence is directly true of production too. Capacity planning done on the test bench transfers one-to-one, and I wrote up the prod side of that arithmetic in what actually fits on the H100 pair.

Without that parity, the test bench would be a different planet. A 48 GB card would force us to test everything in a smaller quantization than we serve, and then every quality verdict would come with an asterisk. Memory parity removes the asterisk.

what is actually different, and how much it matters

Here is the honest comparison, on our hardware:

The bandwidth line is the one that dominates day-to-day feel. Single-stream decode is memory-bound, so roughly half the bandwidth means roughly half the tokens. A 70B-class model in FP8 that decodes at about 41 tok/s on one H100 does about 19 tok/s on the test card, batch of one, on our setup. Not painful for triage. Very noticeable if you forget which box you are ssh'd into.

The NVLink line matters for anything that wants two cards. On prod, TP=2 across the NVLink bridge is nearly free and it is how we serve the biggest models. On the test bench, two cards talk over PCIe, and tensor parallel over PCIe for interactive decode is a sad experiment I ran exactly once. So the test bench is effectively two independent 96 GB machines, not one 192 GB machine. Anything that genuinely needs the paired capacity gets tested against prod directly, off-hours.

The FP4 line goes the other way: the test cards can do something production cannot. Blackwell has native FP4 tensor cores; Hopper does not. That turned the test boxes into our quantization lab, which got its own post: NVFP4 as a pipeline, not an event.

what the test bench actually runs

Four things, in descending order of hours consumed.

• New model triage. A model drops, someone is excited, it goes on a test box within the hour. Does it load in vLLM at all? Is the chat template broken? Does it pass our eval suite? Most models die here, quietly, without ever touching production. This is the single biggest value of the bench: prod stays boring while test absorbs the chaos.
• Upgrade rehearsal. Driver bumps, CUDA minor versions, vLLM releases: all of it lands on a test box first and soaks for a few days under real-ish load. We have caught two vLLM regressions this way this year, one of them a silent output-quality issue that would have been genuinely bad in production.
• Student experiments. We usually have one or two students or juniors attached to the team. They get a test box and near-total freedom. They cannot hurt production because they cannot reach it. The number of weird states a curious student can get a CUDA install into is a renewable resource.
• Quantization work. NVFP4 conversions, calibration runs, eval sweeps. Bandwidth barely matters here; VRAM capacity is everything, and 96 GB is a lot of capacity.

For triage and development you are paying for VRAM capacity, not memory bandwidth. Bandwidth is a production luxury.

the price argument, plainly

An H100 NVL invoiced us in the mid-thirty-thousands, euros, with the enterprise trimmings around it. An RTX PRO 6000 Blackwell was a bit under a quarter of that, and the tower it lives in was under two grand: commodity PSU, commodity cooling, no SAN, no 25GbE, nothing. Both test boxes together, complete, cost noticeably less than one more H100 NVL card alone, before you even price the server to put it in.

For that money we got 192 GB of test VRAM across two isolated failure domains instead of 96 GB in one. Two people can run destructive experiments simultaneously. When a student wedges box one, box two keeps working. A single third H100 in the prod chassis could never have given us that. It would have been more production, which is a different thing than a test bench, and honestly we would have been too tempted to just serve from it.

If you are building your own version of this at smaller scale, the same logic shows up in hardware for local LLMs: capacity gets you in the door, bandwidth is what you pay up for when latency starts paying salaries.

the limitation, stated as bluntly as I can

Performance numbers from the test bench do not extrapolate to production. Not roughly, not with a scaling factor, not at all, in the sense that we forbid ourselves from trying.

Different memory system, different bandwidth curve under batching, different interconnect, different parallelism strategy, sometimes different attention kernels because sm_120 and sm_90 do not always take the same code paths in vLLM. A model that does 19 tok/s single-stream on test might do 41 on prod, or the batched-throughput ratio might be nothing like 2.1x because the H100 saturates differently under concurrency. We measured, we got burned once anyway, and the full story of what survives the trip from test to prod is in what transfers and what doesn't.

So the rule on the whiteboard says: the test bench answers "does it work" and "is it good." Only production answers "is it fast." Correctness and quality transfer. Throughput does not.

One more thing that bit us, for the record: workstation cards in tower chassis are not enterprise gear, and one of the boxes spent its first summer week thermal-throttling because the room it lives in is not a machine room. A forty-euro fan fixed it. The H100s, in their properly ducted DL380 with fans screaming, have never throttled once. You get what you pay for; the trick is knowing which workloads deserve what you paid for.

If your production GPUs are ever running someone's "quick test," you do not have spare capacity. You have a missing test bench. Buy the cheap cards with the big memory. Match the VRAM to prod, ignore the bandwidth, and let production do the only job it should have: serving.