AI hardware ROI for a used-GPU inference build: five-year total cost of ownership
Somebody grabs a stack of secondhand accelerators off a liquidation listing because the price per gigabyte of VRAM looks unbeatable, slots them into a used chassis, and calls it done. Before long a card throws memory errors with no warranty behind it, the resale market has gone soft, and the power bill has quietly passed the API subscription it replaced, same reason every time: the purchase priced the GPUs and stopped there. A used-GPU inference build is second-hand accelerators plus a compatible case, motherboard, memory, PSU, cooling, and a replacement plan for the card that dies, aimed at steady local inference where memory capacity beats new architecture. The alternative isn't one thing: new consumer hardware, refurbished workstation gear, and pay-per-token API compete for the same budget, and when the used cards hold up, VRAM-per-dollar genuinely beats them all. What matters isn't the price tag or tokens-per-second; it's the cost of one quality-gated inference hour over whatever's left of the hardware's useful life, everything else an intermediate measurement.
What the sticker price leaves out
The rule that matters: total cost of ownership only means something over a realistic ownership horizon, including recurring costs and whatever the hardware is worth when you're done with it. Skip either piece and the comparison is fiction. Write it as one relationship:
TCO = capex + energy + support + maintenance + downtime + financing - resale_valueFeed it with current quotes, electricity rates, tax rules, and API prices, in a worksheet you update, not baked into prose as permanent numbers. Prices move fast, the formula doesn't. Model annual cash flows as low, base, and high cases, not one blended average. Split fixed costs (purchase, installation, reserved capacity) from variable ones (electricity, paid tokens, transfer, some support). Human review sits in between: a dedicated operator is fixed, per-exception review scales with how often the model is wrong.
Everything that should be on the invoice
The bill of materials: accelerator, host, memory, storage, network, power delivery, cooling, rack or enclosure, spares, tax, shipping, installation. Subtract anything the organization would have bought anyway: a laptop a developer needed isn't an AI cost, a memory upgrade bought only because the model wouldn't fit is. Run the workload across every candidate route, local, rented, API, and record the same things for each:
- accepted tasks and severe failures
- input, cached input, reasoning, and output volume
- retries, abstentions, and human repair minutes
- queue time, time to first token, completion time, p95
- active, resident-idle, sleep, and unavailable hours
- wall energy and its cooling allocation
- maintenance and incident time
Quality adjustment is where comparisons quietly go wrong. If the cheap route needs two attempts and five minutes of correction, its real cost belongs to the finished result, not the first pass. Run it the other way too: don't price a local model that reliably nails one narrow task against a frontier API at maximum reasoning effort. Not the same job.
Secondhand silicon doesn't run itself
The real risk here: unknown wear, no warranty, undisclosed platform upgrades, higher power draw than spec, downtime, a resale market that may not want the card back in a few years. Build low, base, and high demand curves including daily peaks and p95 latency; a machine can't sit at full utilization and still absorb a request the moment it lands. Batch work soaks up quiet hours, but only if the organization genuinely has that work to run. Meter idle states separately: a workstation sleeping overnight differs from a server keeping models resident for instant response. Shared systems: track queueing and abandoned requests. Edge fleets: multiply update, replacement, backup, and travel time by node count; labor can outweigh electricity. The recurring mistake: assume flawless five-year uptime, then an optimistic resale price nobody's offering. Keep assumptions in a small table, attach a source and date to anything that moves, and don't let decimal precision hide how little you know.
Finding the line where owning wins
Payback month is the first month cumulative discounted benefit clears cumulative cost; ROI over your window is net discounted benefit divided by discounted cost, both resting on a conservative useful life and resale value. Hardware often outlives its economic usefulness: model growth or dropped support can make a working card dead on the balance sheet. Productivity gains need a realization factor too. Ten minutes saved per developer isn't ten minutes of revenue; it counts only if it avoids a hire, cuts outsourced spend, increases what ships, or clears a genuinely binding queue. Run the model once with productivity value at zero. If the purchase doesn't pencil out then, treat it with suspicion.
Compare against hourly rental too. Renting stays useful while demand and hardware shape are uncertain, counting persistent storage, image prep, transfer, minimum billing, setup automation, and the instance somebody forgot to shut down. A short rental benchmark is cheap insurance, the same discipline behind why a flat subscription is the wrong instrument for spiky AI usage: test demand before locking in a cost structure. The durable comparison is discounted ownership cost over the period workload and hardware stay useful, collapsed to operational terms: accepted tasks per month, productive GPU hours, the maximum API cost per task that still loses to owning, or the minimum useful life the hardware needs to hit. Easy to check again later. A recommendation isn't.
Set a retest date. Recalculate when volume shifts materially, a new model changes the quality gate, API prices move, the electricity contract changes, or a component you depend on reaches end of support. None of this is a property stamped onto a GPU; it's a relationship between one workload, one alternative, and a set of dated assumptions, and it goes stale fast. So before you order anything: get a real quote on the cards you're eyeing, price an hour of rental on equivalent hardware, run your workload against it for a week, and write down the break-even number the purchase needs to clear. If you can't state that number, you're not ready to buy.