GPU passthrough for a local LLM virtual machine
Pass the GPU through only when the VM's lifecycle solves a management problem you already have. If it doesn't, skip it. Isolation and reproducibility are real: snapshot a known-good driver stack, rebuild the inference host from a template after a disk dies, keep one workload's crashes off a shared box. None of that shows up in a tokens-per-second number, and it isn't free. You pay in IOMMU groups that won't split the way you want, drivers that never learned to reset cleanly, and memory pinned down before the VM will even boot.
What the IOMMU group actually buys you
The motherboard's bus topology decides this before you get a vote. Devices behind the same bridge or root complex often land in one IOMMU group, and the whole group moves to the guest together or not at all. If you're running a home LLM server sharing one accelerator across several infrastructure roles, that's the reason you're doing this: driver crashes and version upgrades stay in one guest instead of leaking into the host running your network and storage. Before touching a launch flag, write down what number has to move: first-token latency, accepted jobs per hour, resident model capacity, energy per completed task, fewer corrections from whoever reviews the output. Pick one. "Make it faster" doesn't tell you when you're done, or when the change made something else quietly worse.
That's also why the second thing you test is not throughput. Boot the guest, confirm the GPU enumerates, run a real workload, then check whether the numbers land close enough to bare metal to matter. If not, you've learned something about the passthrough path, not the model.
The reset bug that reboots your host
Restart the VM and watch what happens to the card. Consumer drivers were not written with function-level reset in mind, and plenty leave the device in a state the guest can't recover from without a full host reboot. That's the failure worth ruling out first: it turns a routine model swap into an unplanned maintenance window. Local runtimes make this worse in one way: they're stubbornly willing to keep running. Layers get offloaded, memory gets paged, caches get missed, requests queue, and the runtime falls back to a generic kernel rather than fail loudly. That resilience hides a degraded configuration behind output that still looks fine. Read the startup log. Check device placement. Watch the host's counters instead of trusting the guest's self-report, and confirm the optimization is running against the tensor shapes and context length you'll actually use, not the docs' toy example.
Pin everything or watch it drift
Pin the model artifact, the tokenizer, the prompt template, the runtime build, the launch command, and the sampling settings, the same way you'd pin the VM's memory so the hypervisor can't swap it out from under a DMA transfer mid-inference. Test with a small set of real-workload inputs, awkward cases included. Run once cold, unloaded weights and cold storage, if users ever hit that path, then run warm repeatedly to expose cache effects, queueing, memory pressure, and wherever the case temperature settles once it's been busy.
A single total hides where the time goes. Break it into queue wait, model load, prefill, time to first token, decode rate, and the quality check that follows, with peak RAM, VRAM, power draw, and swap tracked too. Token speed alone is diagnostic, not the product. Automation cares about valid completed records per hour. Coding work adds review and fix time. Chat gets judged on p50 and p95 time to first token at a realistic conversation length, not the first exchange of a fresh session. A setup that wins a short, warm, single-request benchmark can still lose the day once you're swapping between models and the prompts get long.
Change one thing at a time unless you're deliberately comparing two finished systems, and keep the actual outputs, not just the metrics: quantization, context compression, and sampling can all make a response faster while quietly making it wrong. If an answer got shorter, check it's still complete. If an extractor got faster, count records that are actually valid, not ones that merely parsed.
Write the retest trigger before you forget why
Operational cost sits next to the performance number, not below it: startup time, upgrades, observability, rollback, plus whether you can rebuild from nothing after a disk dies. A five percent win that depends on an undocumented patch, or someone warming the cache before their shift, is a bad trade for anything shared. Dull setups keep working after you've moved on; clever ones need someone who still remembers why.
The rule holds: use passthrough when the VM's lifecycle solves a real management problem, not because it benchmarks better in isolation. Write that down with the workload, the date, and the reason, plus what should trigger a retest: a new model family, a driver update, longer contexts, another user, a different traffic mix. Skip that step and last quarter's numbers turn into folklore instead of evidence.
Leave headroom once you've hit the target instead of packing the card full. Free memory absorbs the prompt you didn't expect. Spare queue capacity keeps a batch job from starving the person waiting on chat. Thermal and power margin keeps the box behaving the same way at two in the morning as during your benchmark. None of that is about squeezing the last percent out of the card; it's about a system that still behaves itself next month.
So here's the actual next step: restart the guest without touching the host, and see whether the GPU comes back on its own, or whether you've just found your reboot schedule for the next two years.