vLLM on two H100s: the config that serves our whole team

We run one vLLM process on the DL380's H100 pair, and that one process is the LLM endpoint for the entire team: editor plugins, internal chat, CI summarizers, a couple of cron-driven agents. This post is the actual configuration, not a tutorial. If you want the sizing reasoning behind the model choice, that lives in the sizing post; this is the part where it becomes a service somebody pages me about.

The model on the pair right now is a 70B-class checkpoint in FP8, served at tensor-parallel size 2 across the NVLink bridge. We have rotated models under this endpoint several times without clients noticing, which is exactly the property you want.

the flags, and why each one earns its place

• --tensor-parallel-size 2: splits weights and KV heads across both cards. Over the NVL bridge the overhead is small enough that TP=2 is just how the box works, not a tradeoff we think about.
• --kv-cache-dtype fp8: halves KV cache per token, which on our config roughly doubles concurrent capacity. On Hopper this is mature; we could not find a quality regression in our own evals and we did look.
• --max-model-len 32768 is a policy decision disguised as a flag. The model can go longer, but every doubling of max length doubles worst-case KV per sequence. 32k covers real work; people who need 100k+ contexts get routed elsewhere.
• --gpu-memory-utilization 0.92: vLLM preallocates this fraction of VRAM for weights plus KV. We ran 0.95 for a while and got occasional OOMs during CUDA graph capture after restarts. 0.92 has been boring for months. Boring wins.
• --max-num-seqs 64 caps concurrent scheduled sequences so one bulk job cannot starve interactive users into the queue.
• --max-num-batched-tokens 8192 bounds how much prefill work gets mixed into each step. Chunked prefill is on by default in the V1 engine; this knob keeps a giant incoming prompt from stalling everyone else's decode. Before we tuned it, one teammate pasting a whole log file would visibly freeze another teammate's autocomplete.
• --enable-prefix-caching is the sleeper hit. Our editor tooling sends the same fat system prompt with every request, and prefix caching means those tokens are prefilled once and reused. Our cache hit rate hovers around 60 percent on weekdays, and TTFT for cached prompts is a fraction of cold.

the systemd unit

We went with systemd rather than compose because this is bare metal, the GPUs belong to exactly one service, and journald plus systemctl restart is an operational vocabulary everyone on the team already speaks.

[Unit]
Description=vLLM team endpoint (70B FP8, TP=2)
After=network-online.target
Wants=network-online.target

[Service]
User=vllm
Environment=HF_HOME=/data/hf
ExecStart=/opt/vllm/bin/vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic --tensor-parallel-size 2 --kv-cache-dtype fp8 --max-model-len 32768 --gpu-memory-utilization 0.92 --max-num-seqs 64 --max-num-batched-tokens 8192 --enable-prefix-caching --served-model-name team-llm --host 0.0.0.0 --port 8000
Restart=always
RestartSec=10
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target

Details that matter: TimeoutStartSec=900 because loading 72 GB of weights, sharding them, and capturing CUDA graphs takes minutes, not seconds, and systemd's default will kill a perfectly healthy startup. --served-model-name team-llm so clients hardcode a stable alias instead of a checkpoint path, which is what lets us swap models underneath. Weights live on local NVMe; pulling them over the network on every restart is how you turn a 4-minute restart into a 20-minute one. There is a small pile of host-level prep behind this too, IOMMU and NUMA pinning and persistence mode, which got its own writeup in the host tuning notes.

the endpoint everyone eats from

vLLM speaks the OpenAI API, so everything downstream is just a base URL: http://llm.internal:8000/v1. Editor plugins point at it, our internal chat UI points at it, LangChain-ish scripts point at it with model="team-llm". Zero client-side special-casing. New tool onboarding is one env var. This is the strongest argument for vLLM over more exotic servers: the compatibility surface is the product.

We put a dumb nginx in front for TLS and an allowlist. No smart routing, no gateway layer. One box, one process, one upstream.

observability, or how I stopped guessing

vLLM exports Prometheus metrics at /metrics and they are genuinely good. The ones on our dashboard:

• vllm:num_requests_running and vllm:num_requests_waiting: the waiting gauge is the single most honest signal on the box. Nonzero for more than a minute means we are saturated and humans are feeling it.
• vllm:gpu_cache_usage_perc is KV cache utilization. We alert at 90 percent sustained, because past that the scheduler starts preempting sequences and tail latency goes nonlinear in a hurry.
• vllm:time_to_first_token_seconds is a histogram; we alert on p95 above 2 seconds for five minutes. TTFT is what humans perceive as "the AI is slow," far more than decode speed.
• vllm:time_per_output_token_seconds is decode pace; a slow creep here has twice pointed at thermal throttling before anything else did.
• token counters, which feed the per-team usage accounting described in the inference logging post.

Three alerts total: queue depth, KV utilization, TTFT p95. Every additional alert we tried eventually got muted, which tells you what it was worth.

The waiting-queue gauge is the most honest metric on the whole box: it is the exact number of colleagues currently annoyed at you.

what throughput actually looks like

Numbers from our box, hedged accordingly. Single stream, a 70B FP8 on this pair decodes at roughly 40 tokens per second, pleasant for chat. Under real mixed load, aggregate throughput lands somewhere around 2,500 to 3,000 output tokens per second before the queue starts backing up, thanks to continuous batching doing its thing. TTFT for prefix-cached editor requests is a couple hundred milliseconds; cold 20k-token prompts are a few seconds even with chunked prefill. Your numbers will differ. Anyone quoting you unbatched benchmark throughput as capacity planning is selling something.

the single-node tax

Now the honest part. This is one node with no HA, and that shapes everything.

A restart, planned or not, takes about four minutes on our box: weight load, TP shard init, CUDA graph capture, warmup. Four minutes during which every editor plugin in the company times out simultaneously. The first time I did a casual midday upgrade, I learned from Slack within ninety seconds exactly how many people depend on this thing. Upgrades now happen at 07:30 with a heads-up in the channel the day before, like actual operations, because that is what this is.

There is no failover. If the box is down, the fallback is hosted APIs via a client-side cascade, which costs real money and behaves slightly differently, and pretending otherwise would be lying to ourselves. A second node would fix this and is not currently worth two more H100s to us. Ask me again after the next unplanned outage.

If you have a team and a GPU pair, this is the whole architecture I would recommend: one vLLM, one systemd unit, three alerts, a stable model alias, and the humility to schedule your restarts. It is unglamorous and it has quietly become the most-used internal service we run.