AI hardware ROI for an edge or SBC AI fleet: five-year total cost of ownership
An edge fleet of Raspberry Pis and Orange Pis running vision or OCR across scattered physical locations doesn't die from a bad benchmark. It dies from an SD card failing in a warehouse nobody can reach on short notice.
The gear: accelerator, host board, memory, storage, network, power, enclosure, a management node, running always-on sensing, OCR, vision, voice, or narrow automation near the data instead of cloud event processing, a central server, or deterministic sensors that never needed a model. The appeal: low per-node power, a quick local response, less data leaving the building, the instinct behind any local-first cascade. The economic unit is one reliable decision at the right location, tokens and accelerator-hours are just how you get there.
The invoice is the smallest number in this decision
TCO is capex plus energy plus support plus maintenance plus downtime plus financing, minus resale value. Model it as annual cash flows in low, base, and high cases, not one lifetime average. Separate fixed from variable: hardware, installation, and capacity are fixed; electricity, tokens, transfer, and support move with the work. Human review can be either, a dedicated operator is fixed, per-exception review scales with failures.
List every incremental part: accelerator, host, memory, storage, network, power, cooling, enclosure, spares, tax, shipping, installation. Subtract whatever the org would buy anyway. A developer's laptop for ordinary work isn't fully an AI cost. A memory upgrade bought solely to fit a model is.
Every extra node multiplies the parts that aren't silicon
The real risk isn't compute. It's maintenance times node count, storage failures, installation labor, mismatched accelerators, and boards that never leave prototype status. Build demand curves in low, base, and high cases, with daily peaks and p95 latency: a fully scheduled box can't also absorb an interactive request. Batch work fills quiet hours, only if genuinely needed.
Meter idle states separately, sleep and always-resident cost differently. On shared systems, measure queueing and abandoned requests. For an edge fleet, multiply update, replacement, backup, and travel time by node count, the labor of swapping cards can dominate the electricity line. That's the cost people forget to write down.
What a completed answer actually cost you
Quality adjustment matters more than people expect. A cheaper model needing two attempts and five minutes of correction has its real cost in the completed result, not the first draft. A local specialist that reliably clears one narrow task shouldn't be priced against a frontier API at maximum reasoning effort. You're not buying the same thing.
Measure the workload across local, rented, and API routes, recording the same fields for each:
- accepted tasks and severe failures
- input, cached input, reasoning, output
- retries, abstentions, human repair minutes
- queue, first token, completion, p95
- active, resident-idle, sleep, unavailable hours
- wall energy, a share of cooling, maintenance time
Payback has a date, return has a boundary
Payback month is the first month cumulative discounted benefit clears cumulative cost. ROI over the horizon is net discounted benefit divided by discounted cost, both need a conservative useful life and residual value. Hardware can keep running long after the model line it was bought for outgrows it, or after nobody maintains the stack under it. That's obsolescence, not failure, and it still counts.
Productivity gains need a realization factor. Saving ten developer minutes doesn't automatically turn into ten minutes of billable revenue. Count it only when it avoids a hire, cuts outsourced spend, or shortens a binding queue. Run the model once with productivity value at zero: hardware justified only by speculative time savings deserves more scrutiny, not less.
Compare against renting too. Hourly GPU rental earns its keep while demand and hardware shape stay uncertain, and the honest number includes storage, transfer, minimum billing, and the idle instance everybody forgets to shut down. If you've read why a subscription is often the wrong frame for AI cost, the same instinct applies. I wouldn't model financing to the cent here, loan overhead is noise next to labor.
Give the number an expiration date, then check it
The recurring mistake: assuming hardware runs perfectly for five years at full utilization, then keeps an optimistic resale price at the end. Neither half survives contact with real warehouses. Keep assumptions in a small table with a source and a date on anything volatile, an electricity rate, an API price, in a worksheet, not frozen into prose. Prices move faster than architecture, the formula and boundary stay useful.
State the break-even boundary in operational terms: accepted tasks per month, productive GPU hours, maximum API cost per task, or a minimum useful life. That number is easy to check next quarter. A paragraph of reasoning isn't. ROI is a relationship, one workload measured against one alternative, priced with assumptions that carry dates.
Before you sign off, pick a retest date and list the triggers that force an earlier recheck: volume changing materially, a model swap that moves the quality gate, API prices shifting, the electricity contract renewing, or a component hitting end of support. Put the break-even number next to it, so next quarter you're checking a boundary, not guessing why you believed it.