AI hardware ROI for a used-GPU inference build: renting GPU capacity versus buying

A used GPU is nearly always cheaper than the cloud, on paper. That qualifier is the whole article: price the hours a machine spends on billable work, not raw FLOPs, and the deal gets a lot less obvious.

The build here is second-hand accelerators plus a compatible case, board, memory, PSU, cooling, and a plan for when a card dies, because one will. The workload is steady local inference where VRAM matters more than owning last quarter's architecture, against new consumer cards, refurbished workstations, or hourly rental. The pitch: lower price, better VRAM per dollar, if the cards keep working.

What a used card actually costs you, monthly

Charge yourself per quality-gated inference hour over the card's remaining life, not per token or GPU-hour, just the ruler. Work out what renting the job costs: hourly rate times billable hours, plus storage, transfer, and platform fees, then run that against on-demand, reserved, and interruptible pricing on one workload.

Split fixed from variable early. Card, install, reserved capacity: fixed. Electricity, overflow tokens, transfer: variable. A reviewer can be either, dedicated is fixed, exception-triage is variable. I use this split for cost-architecture work; blurring it makes a bad purchase look good.

Price everything incremental, down to the enclosure and install time, then subtract what you'd buy anyway: a laptop isn't an AI cost, a memory upgrade bought solely for a bigger model is. Weight for quality: two tries and a cleanup pass count against the finished result. Don't price a narrow local model against a frontier API running flat out. Different job.

Utilization is the risk you're actually pricing

Used cards carry unknown wear, no warranty, a firmware surprise, real power draw, downtime, and a resale market that won't bail you out. None of that shows on a spec sheet.

Build low, typical, and high demand scenarios, watch daily peaks and p95 latency: a box at full utilization has no room for an interactive request. Batch work can soak up idle time, but only if you'd run it anyway.

Meter idle states separately: a sleeping workstation has nothing in common, cost-wise, with a server holding models resident so nobody waits on a cold load. On shared systems watch queue depth and abandoned requests. Across a fleet, multiply update and replacement time by node count, labor outruns power.

The mistake I see most: comparing a cloud provider's hourly rate to a bare local GPU while forgetting the host and storage the rented instance needs too. Not the same comparison. Write assumptions down somewhere small, date the volatile ones, skip the fake precision.

When the math actually says buy

Payback month is the first month running discounted benefit clears running cost; ROI is net discounted benefit over discounted cost. Both lean on a conservative useful life and a residual value near zero. A card can run fine after the models you want outgrow it, or the driver stack stops updating.

Apply a realization factor to any productivity claim: ten saved minutes a day isn't ten minutes of revenue, it counts only if it avoids a hire, cuts an outsourced bill, or shortens a binding queue. Run the model once with that number at zero. A purchase that clears the bar only on imagined time savings deserves suspicion.

Rent while demand or the hardware landscape is still moving; buy once utilization has held steady long enough to trust it, the same logic as why a flat subscription is often the wrong bet for spiky usage. State the break-even line concretely: tasks per month, productive GPU-hours, a ceiling on API cost per task. Set a retest date too: volume shifts, the quality gate changes, or a component reaches end of support.

So before buying anything, pull last month's usage logs and count the hours the box was doing billable work, not merely powered on. That number decides this, not the spec sheet.