PCIe lanes matter less—and more—than you think

Here's the rule I use: don't touch the PCIe topology until the runtime's own logs prove the interconnect is what's slowing you down. Everything below is the argument for why that rule holds.

My reference machine is ordinary on purpose, two consumer cards in x8/x8 slots with room to breathe between them, the kind of board a normal person owns, not a lab rig. For the broader hardware picture there's a separate rundown on picking that baseline. What people get backwards is where the bus does work. Once a model is loaded and sitting in VRAM, generation is mostly compute, the cards stay busy and the link mostly idles. The bus earns its keep at two moments: loading the model, and splitting it across cards, a decision that lives inside how a runtime picks what goes where. That's it. Two moments. Everything else is noise dressed as a bottleneck.

So before touching a setting, write the job down: chat, code completion, document extraction, an overnight batch. Decide what counts as good for that job, because faster means nothing until you've said faster at what.

measuring the thing you can actually change

The test itself is boring, and that's the point. Check slot wiring, NUMA placement, and peer access as separate line items, and measure prompt processing apart from generation, since they load the bus differently and averaging hides whichever one is hurting you. Feed every run the same input and save the launch command beside the result, because a number you can't reproduce next week was never a result. One warm run tells you almost nothing. Include a cold start, and run long enough that heat or queueing gets a chance to show up.

Only record numbers that could change what you do next: time to first token, prompt-processing speed, generation speed, peak memory, wall power if it matters to you, and whether the task succeeded. Check the median, that's the usual experience, but check the slow tail too, that's the run that quietly kills trust in the setup. Note the soft stuff too, fan noise, responsiveness, setup annoyance, since those costs decide whether it stays in use past week one.

the spec sheet argument nobody needs to have

This is where people waste money, and worse, weeks. Someone passes on a genuinely fine board for lacking x16/x16, or buys one whose second long slot turns out wired at x4 once you read the manual instead of the marketing photo. Local inference is full of configurations that technically run, and loading is not a performance result any more than an answer that looks fine is an evaluation. Pull the runtime logs and OS metrics instead of trusting a requested flag, a flag is an ask, not a fact. When two runs differ, change one variable and say why you expected that to matter, or you have anecdotes with numbers stapled on, not a benchmark.

There's a maintenance cost too, and people underrate it. A fragile five-percent win disappears the moment a driver, model, or runtime updates, often without you noticing it left. A small script, a handful of prompts you care about, and a plain-text results file catches that drift, no monitoring platform required for a single workstation.

spend the lanes on the bottleneck you proved

The rule, then: optimize the interconnect only after confirming the runtime moves enough data across it to make it the bottleneck, test the smallest plausible change first, and stop the moment the workflow clears its latency and quality bar. Leave headroom past that point, because the next model you try, or the background job you forgot was running, will want it. Restraint beats hardware here. The setup worth keeping is the one you understand well enough to trust, not the one with the flashiest isolated number.

What I still don't have a clean answer for is the machine that changes shape over time, a second card bolted on well after the first, or a workload that drifts from pipeline-parallel toward tensor-parallel and starts hammering a link that used to sit idle. I retest when I remember to. I don't have a good trigger for when I should, and I'm not convinced a fixed schedule is the right shape for that problem at all.