AI hardware ROI for a used-GPU inference build: break-even against commercial APIs

Buy the used-GPU box only when the break-even point sits inside its realistic working life, not on faith. That's the whole call. The rest is proving it honestly.

The build: secondhand accelerators, a case, motherboard, memory, PSU, cooling, and a replacement plan. The workload: steady local inference where VRAM matters more than new silicon. The alternative is new hardware, a refurbished workstation, or pay-as-you-go API calls, the bet covered in why subscriptions are often the wrong shape here. Used cards mean a lower price and better VRAM per dollar if they keep working, converted into one unit: a completed task, not tokens or hours.

A single per-token API rate tells you nothing; you need model quality, cached versus fresh tokens, output tokens, tool calls, retries, and the review that catches what slips through. Put today's electricity rate and API price into a live spreadsheet, not prose. Prices move weekly, the formula doesn't.

price the whole rig, not the listing

Break-even task count is local fixed cost divided by the gap between API cost per accepted task and local variable cost, measured per accepted result across both routes, not per attempt. Split fixed from variable: hardware, installation, and reserved capacity are fixed; electricity, paid tokens, and transfer move with volume; a dedicated reviewer is fixed, per-exception review scales with failure rate.

List the bill of materials, accelerator through installation, then subtract anything the organization would buy anyway: a laptop used normally isn't an AI cost, a memory stick bought only so a model fits is, entirely.

Log accepted tasks against severe failures, the token split, retries and repair minutes, latency including the p95 tail, and active-versus-idle hours. A cheaper model needing two attempts and five minutes of correction hands its real cost to the finished result, not the first pass; don't price a local specialist against a frontier API at maximum reasoning effort.

ROI isn't a spec you read off a GPU box. It's a relationship between one workload, one alternative, and a date you wrote the assumptions down.

the hours nobody remembers to bill

Used cards carry real risk: unknown wear, no warranty, lapsing platform support, creeping power draw, unscheduled downtime, a resale market that won't take the card back. Build low, base, and high demand curves instead of one guess: a machine can't run at full utilization and still absorb interactive requests. Batch work fills quiet hours only if it genuinely exists.

Meter idle states separately: a sleeping workstation differs economically from a server holding models resident for instant response. Watch queueing and abandoned requests on shared systems, and multiply update, replacement, and travel time by node count on edge fleets, since labor eats the budget, not electricity.

The recurring error is comparing raw token counts between models that fail and retry differently. Keep assumptions in a small table with a source and date attached, not hidden behind decimal precision.

payback has an expiration date

Payback month is the first month cumulative discounted benefit clears cumulative cost; ROI over your horizon is net discounted benefit over discounted cost. Both need a conservative useful life and residual value: a card can outlive the models it was bought to run.

Productivity gains need a realization factor: ten minutes saved per developer doesn't automatically become ten minutes of revenue. Count it only when it avoids a hire, cuts outsourced spend, or shortens a binding queue. Run the model once with productivity value at zero; if it only pencils out with speculative savings added back, treat that number with suspicion.

Weigh renting too: it earns its keep while demand is uncertain, though true cost includes storage, transfer, minimum billing, and forgotten idle instances, and a short benchmark beats an expensive wrong purchase. Buy only once demand clears break-even inside a defensible horizon, stated as accepted tasks per month, productive GPU hours, or a minimum useful life, a boundary you can revisit unlike a narrative. Retest when volume shifts, the quality gate changes, prices move, or a part hits end of support.

What I still haven't found a clean way to price is resale: what a worn accelerator with no warranty and a shrinking market is worth the day you stop. Put a residual value in the spreadsheet if it helps. I wouldn't lean on it.