Consumer or workstation GPU for local AI?

Two quotes sat open in browser tabs: same core count, same generation, one card carrying ECC memory and a longer warranty, the other missing both and priced well under half. Neither spec sheet said which one to buy. The answer was sitting in what would actually run behind the API once the card went into the box.

what's actually queued behind the request

Professional cards mostly sell operational properties, not proportional speed. You aren't buying a faster answer for the same money; you're buying error-correcting memory, driver support that survives a kernel update without drama, a cooling design meant for a closed chassis rather than an open desk, and sometimes the ability to slice one card into isolated pieces for more than one tenant. None of that shows up on a tokens-per-second chart, and none of it matters if the box in question runs a single chat session for one person a few times an hour.

Write down what's actually queued before you shop. A private chat used a handful of times a day, a coding loop that fires on every save, and a nightly extraction batch can all live on the same box and still want completely different things from it. The chat wants low latency and forgives an occasional slow reply. The batch job wants throughput and total completion time and could not care less about any single request inside it. Buy for the batch job's profile and hand the machine to the chat user, and you've bought the wrong card twice: once in cash, once in a frustrated colleague.

measure the machine you already have

Before any purchase gets a green light, baseline the box sitting under the desk right now. Pick one named model, one fixed prompt set, and write down the exact command that launched the server, flags included. Keep the model artifact and the prompt template with the record, not just a description of them. Those two things drift, and a comparison run six weeks later against something merely "similar" isn't a comparison at all, it's a guess with a chart attached.

Measure the request the way it actually moves through the system, not just the part that's easy to log. Time to first token covers loading and prompt handling. Steady-state token rate covers decoding. Completion time is what the person on the other end experiences, and it can be worse than the sum of the other two if something upstream is queuing. Add peak memory and, when it bears on the purchase decision, wall power. For batch work, count completed valid jobs per hour, since a fast job that produces garbage counts for nothing. For interactive work, count the slow tail a person will actually remember and complain about, not the median that always looks fine.

A short table survives longer than a page of notes:

A benchmark with no decision attached is trivia. A setting nobody owns turns into folklore that outlives the person who configured it.

the expensive mistake and the quiet one

The expensive mistake is obvious: paying multiples more for headroom a single-user desk machine will never touch. It's an easy trap because the machine still works fine either way, and nobody ever gets paged to explain the markup on the invoice.

The quiet mistake is worse, because it never announces itself. Local inference fails softly far more often than it fails loudly. A model partially offloads onto the CPU and everything still runs, just slower. A cache misses without a warning. Swap creeps up over a long session. A queue keeps serving requests whose client already gave up and disconnected. A fallback path quietly sends a request somewhere it shouldn't go, which moves the privacy boundary without anyone deciding to move it. None of these throw an error. Working out whether a slowdown is the card's fault or the cache's fault is exactly the kind of thing local KV cache math is worth sitting with before you blame the GPU.

Watch runtime logs and OS-level counters while a test runs, not just the number the benchmark script prints at the end. Change one variable per run unless you're deliberately comparing two whole configurations against each other. Run it more than once, because a single fast pass could be a lucky sample rather than a real improvement, and there's no way to tell the difference without repeating it. Read the actual output too: a faster wrong answer is not a win. If a change speeds things up but degrades a task that matters, that cost goes in the same report as the gain, not a footnote nobody reads.

Operational simplicity earns its own line item. A clever configuration that needs hand repair after every driver or model update wasn't free just because it was free to build. I'd rather run something boring that comes back from a service file, a container definition, or a short script than something fast that only one person on the team knows how to fix at two in the morning.

the rule that survives the next upgrade cycle

Buy the reliability features, ECC, warranty term, virtualization support, when downtime or density carries a real cost: shared infrastructure serving more than one team, or an unattended job running overnight with nobody watching it. That system has visible limits, and visible limits can be scheduled around, budgeted for, or explained to whoever's asking. Invisible limits just show up as unexplained waiting, then an emergency purchase order, which is worse for everyone including whoever has to approve it.

Stop spending once the workload clears its quality and latency bar with room left to spare. That room isn't sitting idle: it's what covers the day someone pastes in a bigger document, the week the team grows by one more person, the afternoon the office gets hot enough to throttle clocks, or the runtime update that quietly changes how memory gets managed underneath you. For a gut check on what a given tier actually buys before real money changes hands, put your own baseline numbers next to hardware for local LLMs instead of trusting either spec sheet.

Buy the workstation card for the properties you will use, not for the badge on the box.