The sizing math for a 2× H100 96GB pair: what actually fits

Every week someone on the team asks whether model X fits on our H100 pair, and every week I do the same arithmetic on a whiteboard. This post is me writing the whiteboard down so I can send a link instead.

The hardware: two H100 NVL cards in our DL380 Gen11, bridged with NVLink. The spec sheet says 96 GB per card. In practice, call it roughly 94 usable each, so about 188 GB for the pair, and once vLLM takes its 0.92 slice of that (more on why in the serving post), you have about 173 GB to actually spend. Not 192. The gap between the marketing number and the spendable number is where most sizing arguments go to die.

That 173 GB has to cover three things: model weights, KV cache, and a few GB of activation workspace plus CUDA graph buffers. Weights are fixed. Activations are small and roughly constant, maybe 5 to 7 GB across the pair for the models we run. Everything left over is KV cache, and KV cache is what determines how many people can use the box at once. This is the part everyone underestimates.

Weights are a one-time rent payment. KV cache is the utility bill that scales with every user you invite over.

the formula, so we never argue about it again

KV cache bytes per token = 2 × layers × kv_heads × head_dim × dtype_size. The leading 2 is for the K and V tensors. That is the whole formula. Note it uses kv_heads, not attention heads, which is why grouped-query attention was the single biggest gift to people like us who serve models on fixed hardware.

Worked example, Llama 3.3 70B: 80 layers, 8 KV heads (GQA, down from 64 query heads), head_dim 128. At FP16 that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, call it 320 KB per token. A single 16k-context sequence costs about 5.4 GB. One user, one conversation, 5.4 GB. With FP8 KV cache it halves to roughly 2.7 GB per sequence, which is why we run FP8 KV everywhere and stopped feeling guilty about it. I went deeper on this arithmetic in the KV cache math post if you want the derivation for other architectures.

dense 70B: FP16 barely, FP8 comfortably

Llama 3.3 70B at FP16 is about 141 GB of weights. It fits, technically. Subtract 141 and the activation overhead from 173 and you have roughly 26 GB of KV budget, which at 16k context is maybe 9 concurrent sequences with FP8 KV. Nine. On a machine this expensive. We ran this config for about two weeks before admitting it was a waste of silicon.

The same model at FP8 is roughly 72 GB, leaving about 95 GB for KV, which is around 35 concurrent 16k sequences. Quality delta between FP16 and a well-made FP8 checkpoint on Hopper is, in our eval runs, within noise for coding and summarization tasks. Hopper has native FP8 tensor cores, so you are not even trading speed for it; you are gaining speed. In 2026 running a dense 70B at FP16 for production serving is a form of nostalgia.

the MoE middle class is where the value lives

gpt-oss-120b is the pleasant surprise of this generation. It ships natively in MXFP4 at about 63 GB on disk, 117B total parameters with only 5.1B active per token. Weights fit on the pair with over 100 GB to spare, and its KV footprint per token is small, roughly 70 KB at BF16 before you account for the sliding-window layers, which cut it further. On this model the pair is compute-bound long before it is memory-bound. We can hold well over a hundred 16k sequences and the limiting factor is tokens per second, not gigabytes.

GLM-4.5-Air-class MoE (106B total, 12B active) at FP8 lands around 108 GB of weights. That leaves roughly 59 GB of KV, good for something like 25 to 30 concurrent 16k users on our box. Comfortable, not luxurious.

Qwen3-235B: only through the 4-bit door

Qwen3-235B-A22B at FP8 is about 235 GB of weights. Does not fit, full stop. At W4A16 (AWQ or a GPTQ variant) the weights compress to roughly 125 GB, and now it fits with maybe 42 GB of KV headroom. The architecture helps you here: 94 layers but only 4 KV heads, so FP8 KV runs about 94 KB per token, roughly 1.5 GB per 16k sequence. Call it 25 concurrent users. We keep a W4A16 build of it around for tasks where we want the extra reasoning depth, but the 4-bit weight quantization is a real quality tax on this model in a way FP8 is not, so it is not our daily driver.

what does not fit, and please stop asking

DeepSeek-V3 and R1 at 671B: native FP8 is 671 GB, and even an aggressive 4-bit build is 350+ GB. Not on 188 GB, not with any quant you would want to defend in a postmortem. Qwen3-Coder-480B is the same story at around 250 GB for 4-bit. These are 4-to-8-GPU models, or CPU-offload science projects. We did actually try the offload route using the 2 TB of DDR5 in this box, and it sort of works for batch jobs, which got its own writeup in the offload experiments. For interactive use, no.

NVLink makes TP=2 boring, which is the point

Every model above runs at tensor-parallel size 2: each GPU holds half the weights and half the KV heads (4 of Llama's 8 KV heads per card, pleasingly clean). TP requires an all-reduce across GPUs inside every layer, twice per layer, and the sync payload per token is small, kilobytes. That workload is latency-and-bandwidth sensitive in exactly the way NVLink was built for: the NVL bridge gives us 600 GB/s between the cards versus roughly 64 GB/s each direction over PCIe Gen5.

In our measurements the TP=2 overhead over the bridge is small enough that I stopped measuring it, low single-digit percent on decode throughput. On a PCIe-only pair (our earlier test rig) the same config lost us something like 15 to 20 percent, and worse, it made latency jittery under load. If you are buying a two-GPU inference box, the bridge is not optional trim. It is the thing that makes the two cards behave like one 188 GB card.

the table I keep sending people

All numbers approximate, on our hardware, 16k context, FP8 KV cache, vLLM at 0.92 memory utilization:

Two caveats that bit us. First, "concurrent users at 16k" assumes everyone is actually at 16k; real traffic is spikier and prefix caching changes the effective math a lot in your favor. Second, quantized checkpoint sizes vary by a few GB between publishers depending on what stayed in higher precision, so leave yourself 5 GB of slack or your 3 a.m. deploy will OOM during CUDA graph capture. Ours did.

The opinionated summary: on a 2× H100 96GB pair, the sweet spot is a natively-quantized MoE in the 60 to 110 GB range. Dense 70B FP8 is the safe institutional choice. Anything that only fits at 4-bit weights should have to justify itself in your own evals, not in a leaderboard screenshot. And the moment someone says "can we just run DeepSeek on it," send them the table.