2 TB of RAM changes which models you can run

When we specced the DL380 Gen11, the 2 TB of DDR5 was basically a rounding error on the quote. The H100 NVL pair was the line item everyone argued about; the RAM was "sure, max it out, it's cheap relative to the GPUs." Six months in, that afterthought is the reason we can run models that have no business fitting on two 96 GB cards.

The pair gives us roughly 188 GB of usable HBM3 once you subtract driver overhead and the slice vLLM reserves for itself. That is the number everyone plans around. But there are 2 TB of DDR5-4800 sitting behind two Sapphire Rapids sockets, eight channels each, and most days it holds a page cache and some Prometheus containers. That is a waste of a very specific superpower.

what fits where, honestly

Quick inventory of what we actually care about running, with approximate weights-only sizes on our box:

The first surprise: a 235B MoE at 4-bit does fit on the pair. What does not fit is that model at FP8 with real KV headroom, or anything in the 400B+ class at any sane precision. That top row of "no" is where the 2 TB earns its keep. See what actually fits on the H100 pair for the full inventory.

cold experts belong in DDR5

The thing that makes MoE offload work at all is that a Qwen3-235B-A22B-style model only activates ~22B parameters per token: 8 experts out of 128 per layer, plus attention and the shared trunk. The attention stack, embeddings, and router are touched every single token; a given expert FFN might go thousands of tokens without being selected. Hot and cold are cleanly separated. The hardware map writes itself: hot stays in HBM, cold lives in host RAM.

llama.cpp made this almost embarrassingly easy. --n-cpu-moe N keeps the expert tensors of the first N layers on CPU while everything else goes to the GPU, and -ot gives you regex-level control over individual tensor placement if you want to get fussy. Something in the shape of numactl --interleave=all ./llama-server -m qwen3-235b-q4.gguf -ngl 99 --n-cpu-moe 40 is genuinely all it takes to run the FP8-class footprint on this box. KTransformers goes further and runs the CPU-resident experts through hand-tuned AMX kernels rather than treating the host as a dumb parking lot, and vLLM has its own --cpu-offload-gb path, though for pure MoE expert placement llama.cpp's knob is still the most surgical of the three on our hardware.

The experts do not stream over PCIe per token in this setup. They are computed on the CPU where they live. That distinction matters enormously for the next section.

the ceiling is not where you think

PCIe Gen5 x16 tops out around 63 GB/s per direction, and if your mental model is "page experts into HBM on demand," that link is your bottleneck and your token rate is a disaster. Computing cold experts CPU-side dodges the link but runs into a different wall: DRAM bandwidth. Each Xeon 8458P socket gives us maybe 250 GB/s of realistic STREAM-class bandwidth. An MoE step touching ~12 GB of 4-bit expert weights per token puts a hard theoretical ceiling somewhere under 20 tok/s from memory bandwidth alone, before you pay for any actual arithmetic.

Our measured numbers land about where that math predicts. Qwen3-235B-A22B with roughly 40 layers of experts in DDR5 decodes at 6 to 9 tok/s on this box, depending on context length and how the day's NUMA luck goes. Prompt processing is much healthier (a few hundred tokens per second) because prefill batches enough work for AMX to actually stretch its legs.

Offload buys you the ability to run the model at all, not a chat experience.

Eight tok/s is miserable for interactive use and completely fine for batch. Overnight evals, bulk summarization, dataset generation: nobody is watching the cursor blink at 2 a.m. That is exactly the workload we route through the overnight batch window, and it is the only workload where I will defend expert offload with a straight face.

KV cache is the other thing that spills

Long context eats HBM faster than people expect. A 70B-class model at fp16 KV costs roughly 320 KB per token; a single 128k-token sequence is ~40 GB of cache, which on a tightly packed deployment is the difference between serving eight concurrent users and serving two. The arithmetic is worked through properly in the local KV cache math post.

Host RAM gives you a pressure valve. vLLM's --swap-space lets preempted sequences park their KV in host memory instead of being recomputed from scratch, and we run it generously (the default is a few GB; we give it 96). With 2 TB behind us, swapping a monster context out and back over PCIe costs a second or two of latency instead of a full prefill that would cost thirty. It is not free (the swap traffic does ride that 63 GB/s link), but recompute-versus-swap is not a close call at 100k+ tokens.

models staged in RAM, not on the SAN

The least glamorous win: we stopped loading models from storage. The FC SAN gives us maybe 6 GB/s on a good day through both 32Gb links, so a 133 GB GGUF used to mean several minutes of staring at a progress bar every time someone wanted to switch models. Now there is a 900 GB tmpfs, built with mount -t tmpfs -o size=900G tmpfs /mnt/models-hot, holding the four or five models in active rotation, synced from the SAN by a boring cron job.

llama.cpp mmaps weights by default, so a model whose file is already in RAM "loads" close to instantly; the only real cost left is pushing GPU-resident tensors over PCIe, which for a 70B FP8 is well under a minute. Model switching went from a coffee break to a shrug. Even without tmpfs, just letting the page cache keep warm copies gets you 80% of this: 2 TB means the cache never has a reason to evict a model you used yesterday.

the AMX and NUMA fine print

None of the CPU-side numbers above happen on a default build. You want llama.cpp compiled with AMX support so the int8 and bf16 tile units on Sapphire Rapids do the expert matmuls. On our box it is worth roughly 2x on prefill versus plain AVX-512. And you must think about socket topology: expert tensors interleaved across both sockets' memory while the compute threads sit on one socket will quietly halve your effective bandwidth. We burned an afternoon on exactly that before rereading our own dual-socket NUMA notes. Interleave everything or bind everything; the miserable middle ground is the default.

where I landed

Host memory makes the big models possible, not fast, and possible is a real product: we run FP8-precision 235B evals and 400B-class batch jobs on a machine whose GPUs should top out around 188 GB, and the marginal cost was zero because the RAM was already in the chassis. My rule after six months: anything interactive runs fully resident in HBM, full stop, and anything that tolerates single-digit tok/s gets to borrow the big model via offload at night. If you are specing a similar box, do not economize on DIMMs to afford a slightly better GPU — the GPU decides how fast you go, but the RAM decides what you are allowed to attempt.