AI hardware ROI for a shared team GPU server: the utilization curve

One multi-user Linux host, one or more GPUs, server memory, storage redundant where needed, a network link, and somebody on call: that's the shared team GPU server, spending its day on coding-assistant calls, RAG lookups, extraction jobs, evaluation runs, and whatever batch work got scheduled overnight.

The alternative is always commercial APIs, or rented inference sized to the same traffic. Owning the box buys pooled utilization, control over where data lands, predictable model versions instead of whatever shipped this week, and capacity you don't renegotiate every spike, the case for a local-first cascade over a hosted API. None of that matters unless it wins on the number that counts: one accepted task at the latency users tolerate, not tokens, not accelerator-hours.

Here's the rule that decides everything else: fixed cost divides across the hours the box does something useful, and idle time still burns capital, and often power, whether anyone used it or not. Turn that into a number with your own quotes, electricity rate, and API price sheet, not whatever was true when someone wrote about it. Architecture ages slowly. Prices don't.

the parts list nobody itemizes

Effective cost per hour is annual fixed cost divided by productive GPU hours, plus whatever varies with the work. That's the whole formula, and almost nobody separates the two before buying.

Fixed costs get paid whether the box runs one job or ten thousand: accelerator, host, memory, storage, network, power and cooling, rack or enclosure, spares, tax, shipping, installation. Variable costs move with usage: electricity, paid API tokens, data transfer, some support. Human review sits in the middle: a salaried reviewer is fixed, someone pulled in only when output needs fixing is variable, since it only shows up when the model fails.

Strip out anything the organization would buy anyway before the bill of materials counts as honest: a developer's laptop doing ordinary work isn't an AI cost; a memory upgrade bought specifically so a model fits in RAM is.

Instrument the real workload across every route, local box, rented instance, API, logging the same fields for each: accepted tasks and severe failures, the input/cached-input/reasoning/output split, retries, abstentions, human repair minutes, queue time, first-token time, completion time, p95, hours active versus resident-idle versus asleep versus unavailable, wall energy and cooling share, and time lost to maintenance and incidents.

Quality has to enter the price. A cheap model needing two attempts and five minutes of human fixing has an effective cost belonging to the finished result, not the first, misleadingly cheap generation. A local model that reliably nails one narrow job shouldn't get benchmarked against a frontier API at its most expensive reasoning setting on the same task, because that isn't the job either of them is doing.

the hour nobody bills is the one that sinks the math

The failure mode was never the hardware. It's queues, idle capacity, the labor of keeping one host alive, an outage on your only host, and demand outgrowing the shape you bought for. Model demand three ways, slow-month, typical-month, peak-month curves, holding p95 latency fixed throughout: a shared box can't run at full occupancy and absorb an interactive request without somebody waiting. Batch work can soak up quiet hours, but only if that work was needed, not because the GPU was free.

Idle states aren't interchangeable: a workstation that sleeps overnight differs economically from a server keeping several models resident so the first request skips a cold-start tax. On a shared box, watch queueing and abandoned requests, that's where demand breaks the model. An edge fleet means multiplying update time and replacement parts by node count, plus backups and travel to reach every site, and operational labor can outweigh electricity once that math gets done honestly.

The single most common mistake is dividing by 8,760 theoretical hours a year, as if the box were owed full uptime regardless of demand. Keep every assumption in a small table with a source and date attached to anything likely to move, and resist writing costs to two decimal places when the underlying number is a guess dressed up as precision.

Payback month is the first month cumulative discounted benefit clears cumulative cost; ROI over your horizon is net discounted benefit divided by discounted cost, both resting on a useful life and residual value you aren't inflating, since hardware often outlives the model ecosystem or the vendor's support. Productivity gains need a realization factor: ten saved minutes isn't automatically ten minutes of revenue, so count it only if it avoids a hire, cuts outsourced spend, raises output, or shortens a binding queue, then rerun with that term at zero. A purchase living only on speculative savings deserves extra scrutiny.

Weigh rental against both the API and the outright purchase: hourly GPU rental earns its keep while demand and hardware shape are uncertain, so long as you count persistent storage, image setup, transfer, minimum billing, setup automation, and the instance someone forgot to shut down. A short rental trial before committing capital can save you from an expensive wrong purchase for a fraction of what it costs.

State the boundary in numbers you'll trust in six months: accepted tasks per month, productive GPU hours, max API cost per task that flips the decision, minimum useful life. That boundary you can revisit; a narrative recommendation gets forgotten in a slide deck. Set a retest date and name the triggers: material volume change, a model shift that resets quality, API price moves, a new electricity contract, a component hitting end-of-support. ROI was never a property stamped onto the GPU, it's a relationship between one workload, one alternative, and a set of dated assumptions.

Buy for the utilization you can actually sustain, not the peak you're afraid of someday missing, and accept the tradeoff: on the day demand spikes past your curve, you'll be the one watching a queue grow while whoever stayed on the API that month pays a little more and moves on. That's the headroom you're giving up on purpose, for a steady-state cost the API was never going to match.