Model parallelism across home GPUs

Two GPUs sharing a motherboard do not average their bandwidth. The slower card, and the link between them, sets the pace for anything that crosses it. That's true before you touch a launch flag, and it's why doubling the cards rarely doubles the throughput.

Split a model's layers across two mismatched consumer cards and you've built a relay race: each device does its slice, hands off activations, and the pipeline waits regardless. Capacity goes up immediately; speed is a separate, ongoing negotiation. Decide what you're chasing first: first-token latency, jobs per hour, resident model size, fewer corrections downstream, not just faster. If the cards aren't bought yet, what a given pairing can hold is worth reading first.

the handoff is the tax, not the raw compute

Pipeline parallelism leaves one card idle while the other finishes its slice unless you overlap batches; tensor parallelism makes every layer pay a synchronization cost across whatever link connects the cards, plain PCIe on most home rigs. Before comparing split strategies, pin what you're testing against:

• exact model artifact and quantization
• tokenizer and prompt template, unchanged run to run
• runtime build and launch command, written verbatim
• sampling settings, fixed
• a small set of real inputs, awkward cases included

Run once cold, unloaded model included, then warm long enough to expose cache effects and thermal throttling. Test layer splits against tensor splits at context lengths you'll use, not whatever fits a screenshot. The offload arithmetic gets fiddly once KV cache growth per device is in the mix; get it wrong and the tokens-per-second drop looks like a mystery unrelated to spec sheets.

what the stopwatch has to catch (multigpu)

Record phase timings, not one total:

queue wait
model load / activation
prefill
time to first token
decode rate
peak VRAM, RAM, swap
quality pass / retry

Raw token speed is diagnostic. What ships is completed records per hour for automation, review time for coding, or p95 first-token latency for chat at realistic conversation length. Change one variable per test, keep the outputs, and note that quantization and context tricks can speed a run up while changing the answer.

the config that lies to you while it runs

The failure that gets people is assuming compute adds up linearly past batch one. Local runtimes keep going so willingly they'll offload a layer to system RAM, miss a cache, or drop to a generic kernel rather than fail loudly, so a split that looks correct can still run degraded. Read the logs. Check device placement, and weigh what a bad config costs beyond the benchmark: startup, upgrades, rebuilding after a disk dies. A gain needing an undocumented patch or manual warming is a bad trade for anything you're not babysitting.

Fit the model with the second card first, then optimize the split against measured numbers, and write down what should trigger a retest: new model family, driver update, longer context window. Leave headroom after hitting the target, since prompt variance and thermal spikes eat whatever margin you didn't reserve, and every layer here stays inspectable, unlike a vendor's number. What I still don't have is a clean rule for the point where interconnect contention on a full pipeline stops being a bit slower and starts meaning the bigger single card would have been the smarter buy.