The server around the GPUs: DL380 Gen11 host tuning notes

Everyone wants to talk about the GPUs. Nobody wants to talk about the server they're bolted into, which is a shame, because a decent chunk of our inference performance came from unglamorous host-side work on the DL380 Gen11: BIOS screens, numactl invocations, and an honest reckoning with what 32Gb Fibre Channel actually delivers. These are the notes I wish I'd had on day one.

The box, for context: 2× Xeon Platinum 8458P (44 cores each, Sapphire Rapids), 2 TB of DDR5, 2×25GbE, dual 32Gb FC to the SAN, and the two H100 NVLs bridged with NVLink. A serious machine that will happily run slow if you leave everything on defaults.

NUMA is not optional on this chassis

This is a two-socket machine, which means it is two computers wearing one sheet-metal trench coat. Each H100 hangs off the PCIe lanes of one specific socket. Memory attached to the other socket is reachable, but every access rides the UPI links between sockets, and that shows up as real latency and real bandwidth loss on host-to-device transfers.

The rule we enforce: a process that talks to GPU0 runs on socket 0's cores, allocates from socket 0's memory, and receives its network interrupts on socket 0. Same for GPU1 and socket 1. In practice that's numactl --cpunodebind=0 --membind=0 wrapped around the serving process, and checking nvidia-smi topo -m once to confirm which GPU lives where. On our board GPU0 is local to node 0 and GPU1 to node 1, but verify on yours. Assuming is how you lose 10% quietly.

The subtle part is interrupts. irqbalance will cheerfully spray your NIC interrupts across both sockets, so we disabled it and pinned the 25GbE queues' IRQ affinity to the socket local to each serving process. I went deeper on the measurements (including the cross-node penalty we clocked before fixing this) in the NUMA writeup. Short version: the fixes were worth roughly 8–12% on token throughput on our hardware, approximately, and cost nothing but an afternoon.

Tensor-parallel jobs that span both GPUs are the exception. They inherently cross the socket boundary, and NVLink between the GPUs saves you from the worst of it. Don't agonize there. Agonize over the single-GPU serving processes, which is what runs 12 hours a day.

the BIOS screens nobody reads

Out of the box, HPE ships a balanced power profile that is perfectly reasonable for a file server and mildly hostile to latency-sensitive inference. Two changes mattered.

First, the power profile. In RBSU (or through iLO), we set the HPE Power Profile to Maximum Performance. Second, C-states. Deep package C-states save real electricity, but the wake latency lands exactly where you don't want it: time-to-first-token on a request that arrives after a quiet moment. With deep C-states enabled we saw occasional TTFT spikes in the tens of milliseconds that vanished when we limited how deep the cores sleep. For a box that serves interactively all day, we accepted the idle power cost. Your accountant may vote differently; at least make it a decision rather than a default.

Now, the part the spec sheets omit: sound. A 2U server with two 400W GPUs under sustained load is not office furniture. Under a full batch run the fans on this thing produce a noise I would describe as "small regional airport." It lives in a rack in a room with a door now, and that door is a load-bearing part of the architecture. Before the door, people held meetings elsewhere.

Related: watch for thermal throttling before blaming software. nvidia-smi dmon will show you clocks and temperature per GPU every second, and DCGM records throttle reasons continuously. We've stayed clean since sorting airflow, but during one heatwave week in June the intake air crept up and clocks sagged approximately 6%. The graphs caught it before anyone's benchmark did.

A two-socket server is two computers wearing one trench coat. Treat it that way and the GPUs get faster without you touching a single line of inference code.

storage: the SAN is fine, the first load is not

Models live on the FC SAN, which was the pragmatic choice: it's where our storage, snapshots, and backup discipline already live. But 32Gb FC is roughly 3–4 GB/s of realistic sequential throughput per link, and model files are enormous. Our biggest deployed model is about 140 GB of FP8 safetensors; a true cold load off the SAN takes several minutes end to end, with weight loading and engine initialization on top of the raw transfer. The first time we did an emergency model swap mid-morning, those minutes were extremely long minutes.

The saving grace is the 2 TB of RAM. Linux page cache means the second load of the same files is dramatically faster: the weights come from memory, not the SAN, and the reload is dominated by vLLM's own startup rather than I/O. With 2 TB we effectively cache every model we regularly run, plus the OS never feels pressure to evict. This is the least appreciated benefit of specifying a stupid amount of RAM.

For the true cold-start case we're partway into a local NVMe staging tier: a scripted rsync of the active model set from SAN to a local NVMe pair, so cold loads read at NVMe speed and the SAN remains the source of truth. It's a cache with one job. If you're building fresh, I'd put local NVMe in the spec from the start and stop thinking about it.

network: mostly boring, one surprise

The 2×25GbE bond sounded like a potential concern and mostly isn't. Serving tokens is comically light on network: even a busy day of team traffic through the vLLM server is a few Mbit/s of JSON. You will never bottleneck token serving on 25GbE. Stop worrying about it.

The surprise was the embedding backfills. When the overnight queue re-embeds the doc corpus, the fetch stage pulls documents from the corpus store fast enough to flatten one 25GbE link for stretches. We measured sustained periods near 23 Gbit/s on our hardware. Fine at 02:00, and a good reason those jobs are banished to the night window in the first place; the scheduling logic is in the batch queue writeup. If you run backfills during business hours over shared links, your colleagues will find you.

monitoring: feed the Prometheus you already have

Nothing novel here, deliberately. DCGM exporter for the GPUs (utilization, memory, temps, throttle reasons, per-process accounting) plus node exporter for the host, both scraped by the same Prometheus that already watches everything else, dashboards in the existing Grafana. The request-level logging story is separate and covered in the inference logging notes; the host layer is just two exporters and twenty minutes of dashboard work.

The one dashboard panel that has earned its keep is throttle reasons over time. GPU problems announce themselves there before they show up anywhere users can see.

power is a line item, not a footnote

Plan the PDU for the real number. Each H100 NVL is rated at 400W, the two Sapphire Rapids sockets plus 2 TB of DDR5 plus fans at full song add up fast, and our measured wall draw under combined GPU and CPU load peaks around 1.5–1.6 kW. Provision the PDU and the circuit for that, not for the idle number, and check what else shares the circuit before the first big batch night rather than after.

And the electricity belongs in your cost model. At roughly 1.3 kW average under load, 24/7 operation is on the order of 950 kWh a month. At our tariff that's a triple-digit monthly line item that meaningfully moves the $/token math for a single-box operation. The /local-cost page carries power explicitly for exactly this reason. Amortized hardware minus electricity is a fantasy number.

the limitation, stated plainly

It is one box. That's the whole caveat, and it's a big one. Every BIOS update, every iLO firmware patch, every kernel bump that touches the NVIDIA driver is a maintenance window for the entire team's AI tooling. There is no rolling upgrade, no failover, no draining traffic to the other node, because there is no other node. We batch host maintenance into announced Friday-evening windows, snapshot before firmware, and keep the previous kernel bootable. It's fine. It's also the strongest argument I know for a second, smaller box someday — not for capacity, for the ability to touch one machine without turning off the lights.

If your GPUs feel slower than the reviews promised, the odds are decent the server is the reason. Spend the unglamorous afternoon: check nvidia-smi topo -m, pin your processes, fix the power profile, watch the throttle-reason panel for a week. None of it is clever. All of it compounds, and unlike the next model release, it's entirely under your control.