AI hardware ROI for a used-GPU inference build: pricing risk and downtime
Buy the used cards when the expected cost of a failure, not the sticker price on the listing, beats every other way to get the same inference hour. That's the rule. Nothing else here is philosophy, it's just the arithmetic you owe yourself before you trust that rule enough to wire money to a stranger selling take-it-or-leave-it hardware.
The build in question is second-hand GPUs dropped into a compatible case, motherboard, memory, PSU, cooling, and a spares plan you actually intend to use. The workload is steady local inference, the kind where VRAM capacity buys you more than whatever the newest silicon generation is claiming this quarter. The competing routes are new consumer hardware, a refurbished workstation bought from someone who stands behind it, or paying per token through an API. What you're really pricing is not tokens and not GPU-hours, it's one quality-gated inference hour delivered over whatever's left of the card's useful life, and everything downstream of that has to trace back to it.
What you're actually pricing when you price a GPU
Start the model with cash, not vibes. Fixed costs are the purchase, the install, and any reserved capacity, you pay them whether the machine runs one job or a thousand. Variable costs are electricity, paid tokens, transfer, and the slice of support that scales with volume. A dedicated operator watching the rig is a fixed cost; an engineer pulled in only when a job fails is variable, and that distinction changes which number you're allowed to average across a month. Write down every incremental line in the bill of materials: accelerator, host, memory, storage, network, power, cooling, enclosure, spares, tax, shipping, installation, then subtract whatever the business would have bought regardless. A laptop your team already owns is not an AI cost. A memory stick bought solely so a model fits in RAM is. That subtraction discipline is most of what makes a cost architecture actually hold up under scrutiny instead of just looking tidy in a spreadsheet.
The failure math the auction listing won't show you
The number that actually matters is expected annual risk: probability of failure, times repair cost plus outage hours times what an hour of downtime costs the business. Used cards carry risks a warranty normally absorbs for you: unknown wear, no support line, driver or platform requirements that shift under you, higher power draw than the spec sheet implies, and thin resale liquidity if you need to unwind the bet. List the specific ways a component, a provider, a model, or an operator can fail, then go test recovery time instead of guessing it, because your guess will be optimistic. Quality matters as much as uptime here: if the cheap route needs two attempts and five minutes of human correction, that cost belongs to the finished, correct answer, not to the fast first draft nobody can actually use.
Utilization is a curve, and the curve lies if you let it
Model low, base, and high demand, and include the daily peak, because a machine pinned at full utilization has no room left to absorb an interactive request that shows up mid-batch. Idle states are not free: a box that sleeps overnight has different economics from one holding several models resident so it can answer instantly, and batch work only counts toward utilization if the business genuinely needs that batch work done, not because it's convenient filler. Shared systems need queueing and abandonment measured directly; fleets need update, replacement, backup, and travel time multiplied by node count, and operational labor routinely outweighs the electricity bill once you add it up. The recurring mistake is pricing enterprise-grade redundancy into the option you're skeptical of while quietly assuming the cheap option never breaks. Keep every assumption in a small, dated, sourced table instead of hiding the uncertainty behind decimal places that make it look more certain than it is.
Uptime you didn't pay for is a bill with the due date left blank.
When to recheck the number you just calculated
Payback is the first month cumulative discounted benefit clears cumulative cost; ROI is net discounted benefit over discounted cost across a conservative horizon, and both need an honest residual value, because hardware can still power on long after the models you need have outgrown it. Productivity gains need a realization factor: minutes saved only count if they avoid a hire, cut outsourced spend, raise deliverable output, or shorten a queue that was actually binding, so run the whole model once with that value set to zero and see if the purchase still survives on its own. Price rental too, not just buy-versus-API, the way anyone who's weighed why a flat subscription rarely matches a spiky workload would expect: include storage, image prep, transfer, minimum billing blocks, and the instance you forgot to shut down. A short rental trial before the purchase is cheap insurance against an expensive mistake. State the break-even as an operational boundary: accepted tasks per month, productive GPU-hours, or a maximum tolerable API cost per task, because a boundary is something you can check against reality later, and a narrative recommendation is just an opinion with better formatting. Set a retest date and the triggers that force an early one: volume moves, the quality bar moves because the model changed, API prices move, or a major part reaches end of support.
I still buy the used cards for this kind of workload, and I accept, on purpose, that I'm trading the vendor's warranty and support queue for VRAM per dollar, which means the downtime risk the spreadsheet above is measuring now sits on my desk, not someone else's.