MIG-slicing one H100 so the whole team stops fighting over it

For the first few months, GPU 1 on the DL380 was allocated by Slack message. "Anyone on GPU 1?" followed by silence, followed by two people's jobs OOMing each other at 3 p.m. We tried a lock file. We tried a shared calendar. Both were exactly as effective as you'd guess with seven engineers and one 96 GB card.

MIG fixed it, and the fix has been boring in the best possible way. GPU 0 stays whole. It runs the big model behind our shared vLLM endpoint and nobody touches it. GPU 1 got carved into hardware-isolated slices, one per standing workload, and the Slack channel went quiet.

the layout we settled on

An H100 NVL supports up to seven MIG instances. The profile names follow a compute-slices-dot-memory shape: on our card the useful ones are in the neighborhood of 3g.47gb, 2g.24gb, and 1g.12gb, though the exact names vary by SKU and driver, so trust nvidia-smi mig -lgip on your own box over anything I write here. Our carve uses all seven compute slices:

• 3g.47gb: the dev playground. Free-for-all, no SLA, wiped and reclaimed without warning. Big enough for a 32B AWQ model or a serious quantization experiment.
• 2g.24gb: the CI model. A small fast instruct model in FP8 that answers commit-hook and pipeline checks in a few hundred milliseconds.
• 1g.12gb: the embedding server. Embedding models are tiny; this slice runs at maybe 15% utilization and nobody cares, because it costs one-seventh of a GPU.
• 1g.12gb: Whisper. large-v3 plus VAD fits with room to spare, feeding the transcription pipeline.

Four standing services, one card, zero scheduling conversations. The playground being the biggest slice was deliberate: the whole point of having the box is that people experiment, and experiments are exactly the workloads you most want walled off from production.

setting it up is three commands

Enable MIG mode on the target GPU with sudo nvidia-smi -i 1 -mig 1. This usually wants the GPU idle and may want a reset. Then create the instances in one shot, something like sudo nvidia-smi mig -i 1 -cgi 3g.47gb,2g.24gb,1g.12gb,1g.12gb -C, where -C also creates the compute instances so you don't have to do it as a second step. Then nvidia-smi -L lists every instance with a MIG- prefixed UUID, and those UUIDs are how everything downstream addresses a slice.

Plain Docker deployment is one flag: docker run --gpus '"device=MIG-GPU-4f8a...-1c02..."' vllm/vllm-openai and the container sees exactly one small GPU, nothing else. Under Kubernetes the NVIDIA device plugin in mixed strategy exposes each profile as its own resource type. A pod requests nvidia.com/mig-1g.12gb: 1 and the scheduler does the rest. We run plain Docker with a compose file that pins UUIDs; it is unfashionable and it works.

how MIG gets along with vLLM

Perfectly well, with one rule: one engine per instance, always. A MIG slice presents as a complete standalone GPU with its own memory and its own SMs, so vLLM neither knows nor cares that it is a fraction of an H100. What you cannot do is tensor-parallel across slices: no P2P, no shared anything, and NVLink is entirely irrelevant to MIG instances, including the bridge to GPU 0. NCCL jobs spanning multiple instances are similarly a dead end. If a workload needs more than 47 GB or more than three compute slices of throughput, it does not belong on the sliced card. That is deliberate: the sliced GPU is for small permanent things, and the pressure to keep them small is healthy.

Throughput scales about how the slice math suggests. Our 1g instances deliver very roughly one-seventh of the card's compute, which sounds bad until you notice that an embedding model or Whisper was never going to use more anyway. Before MIG those workloads were squatting on a whole H100 at 4% utilization. The utilization graph after the carve is the most satisfying chart I made all quarter.

the isolation is real

This is the part that separates MIG from every software-level sharing scheme. Each instance gets dedicated SMs, a dedicated slice of L2, its own partition of memory bandwidth, and hard fault isolation. When someone's playground experiment OOMs (and it OOMs weekly), the embedding server two slices over does not feel it. Latency on the CI model does not wobble when Whisper is chewing through an hour of audio. We measured p99 on the embedding endpoint before and after a deliberately abusive stress test in the playground slice: it moved by roughly nothing. Time-slicing and MPS cannot make that promise. MIG makes it in silicon.

A MIG slice is the only form of GPU sharing where a noisy neighbor is a physical impossibility, not a monitoring problem.

the fine print that bit us

Three things, in ascending order of how much they annoyed me.

First, reconfiguration requires draining the GPU. Every process on every instance has to stop before you can change the geometry. In practice the layout stays close to frozen. We treat re-carving as a maintenance-window event, twice so far in six months. Design your layout assuming you will live with it for a quarter.

Second, MIG geometry does not survive a reboot by default. We learned this on our first kernel patch: the box came back up, GPU 1 was whole again, and four services crash-looped against UUIDs that no longer existed. The fix is nvidia-mig-manager or a systemd unit that re-carves at boot and, critically, regenerates the UUID pins, because the instance UUIDs change on recreation. Ours templates the compose file from nvidia-smi -L output at startup. Ugly, reliable.

Third, observability. Plain nvidia-smi will not give you per-instance utilization. The utilization column just shrugs at you. You need DCGM with MIG-aware metrics to see which slice is actually busy, and wiring that into our dashboards took longer than the entire MIG setup itself.

the opinionated close

If you have one big GPU and more than three people, MIG is conflict resolution implemented in hardware, not an optimization you bolt on later. Keep one card whole for the workload that genuinely needs all of it, slice the other, make the biggest slice the playground, and automate the reboot re-carve before it burns you rather than after. The MoE offload tricks get the blog traffic, but the MIG carve is the change my coworkers actually thanked me for.