Mixing GPU generations in one inference host
So no, two GPU generations in the same box doesn't hand you a bigger GPU. It hands you two GPUs that disagree about speed, and a runtime that has to arbitrate. Whether that becomes a working pool or a slow card holding a fast one hostage comes down to three things: kernel support for both architectures, link bandwidth between them, and whether anything watches the balance of work each carries.
A pool of unequal cards is not a bigger card. It's two cards with an opinion about whose turn it is.
The runtime doesn't know your cards are different sizes
Say you've got an older 24 GB card next to a newer, smaller accelerator: more VRAM and more compute on paper, a pool. In practice it's a pool only if the runtime gives each card proportional work. The common mistake: treating them as equals, same batch size, same layer split, throttling the faster card to the older one's pace. Local runtimes hide this well: they offload layers, page memory, eat cache misses, queue requests, or fall back to generic kernels, invisible unless you go looking. Before changing anything, write down what needs to move: first-token latency, jobs accepted per hour, resident model capacity, energy per task, how often a human fixes output after. "Make it faster" isn't a target. It doesn't tell you when to stop.
Freeze the setup before you freeze anything else
None of what follows means anything until you pin the stack: model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Pull a small set of real workload inputs, awkward cases included. Run once cold if users hit that path, then warm long enough for cache effects, queuing, memory pressure and heat to show. Check next whether both cards agree on which kernels they're running, not how evenly you split weights: a driver mismatch or silent fallback wrecks the split before load balancing matters.
Clock the stages, not the run
What you want from a run is a timeline, not one number: queue wait, load or activation, prefill, time to first token, decode rate, completion time, peak memory and power per card, quality pass or retry. Raw tokens per second is a fine diagnostic and a poor scoreboard. For automation, count valid completed records per hour. For a coding assistant, count review and correction time: a fast wrong answer costs more than a slow right one. For chat, watch p50 and p95 first-token latency across conversations that get long. A split that wins short and warm can lose once users start swapping models and stretching context, where the KV cache math bites.
Same knob, one variable, save the output
Change one variable at a time unless you're comparing two finished configurations end to end, and keep the outputs, not just the metrics. Quantization, context compression, sampling tweaks and model swaps can make a run look faster while changing the output, so the quality gate runs on every candidate. Shorter answer, check it's complete. Faster extractor, count records that validated, not ones that merely parsed. Weigh operational cost too: startup, upgrades, observability, rollback, rebuilding after a disk dies. A five percent win needing an undocumented patch or hand-warmed cache is a bad trade for anything you don't want to babysit. Boring configurations are the ones still running next year.
Split by the clock, not the spec sheet
The rule that's held up for me: split by measured stage time, and consider not merging the cards into one pool at all. Two separate resident services, multi-model VRAM juggling with a size mismatch on top, often beats a unified split, because it sidesteps one card waiting on the other. Write the conclusion into the result file with workload, date and reasoning, plus what should trigger a retest: new model family, driver update, longer contexts, another user, different traffic mix. Skip that and benchmark numbers turn into folklore. Leave headroom past target: free memory for the unexpected prompt, spare queue so a batch doesn't starve an interactive user, thermal margin for running unattended at night. None of this needs either card at a hundred percent, just the cheapest setup that clears the bar. What I give up: one throughput number for a slide, and whatever peak either card could hit alone. What I get: two boring services and answers I can stand behind. That's the trade. I'd take it again.