NVIDIA or AMD for a local inference box?

Here's the rule, stated before I've earned it: buy whichever vendor's software stack already covers the models and quantization formats you actually run, and budget real hours for keeping that stack alive, because the card itself is the smallest cost in the whole exercise. Everything below is just me working out why, and not memory bandwidth, and not a peak tokens-per-second number pulled off someone else's launch-day slide.

Three workloads, one box, three different appetites

Picture a Linux workstation carrying a private chat interface someone opens a couple times an hour, a coding loop firing constantly through the work day, and a nightly batch job pulling structured data out of a pile of documents while nobody's watching. Same GPU, same driver stack, three different tolerances for latency and memory. The coding loop cares about time to first token above everything, because someone is watching the cursor blink. The nightly job wants completed valid jobs per hour instead, quietly failed validation counted honestly too.

Before reconfiguring anything, write down a baseline: one named model, one fixed prompt set, the exact server command, artifact hash included. Six fields per run: artifact, runtime, and launch flags; workload and fixed input; cold start next to warm start next to p50 and p95; peak memory and wall power; outputs failing a quality check or quietly refused; a decision with owner and retest date. Skip the fixed input and you'll swear in six weeks some setting changed something, with no way to check.

The failure that never throws an error

This is where vendor choice bites, rarely as an error message. CUDA had a head start and covers most quantization formats and attention kernels within days of a new paper. ROCm has closed a lot of that gap, but coverage stays patchier in the places you don't check until after you've bought the hardware. A missing kernel doesn't fail loudly, it falls back to something slower, or a runtime quietly offloads part of a layer to the CPU (worth its own look: how ollama decides what goes where), or a fused attention path turns out not to be. The model still answers, just slower, with a memory footprint creeping until swap kicks in, or a queue quietly holding disconnected requests. None of that trips an alarm. Get a fallback wrong and a request ends up served somewhere your privacy boundary never assumed, its own kind of expensive.

The fix is boring: watch runtime logs and OS counters while a test runs, not the summary line at the end. Change one variable at a time unless deliberately comparing whole configurations. Run it more than once; a single fast pass proves nothing durable. Read the actual output, a faster wrong answer isn't a win you bank. If a change buys speed and costs correctness somewhere, write down both numbers, not just the flattering one.

Pricing the hours instead of the card

Operational simplicity is a real line item, not a nice-to-have. A clever optimization needing hand-patching after every driver update isn't saving you anything, it's borrowing against next quarter. Prefer the boring version: a service file, a container definition, a short script, reproducible from a clean checkout rather than half-remembered fiddling. Keep raw benchmark artifacts out of the report if you want, but keep hashes and exact commands, so a rerun six months later means what it meant the first time. The fuller checklist upstream of this is a separate piece: what actually goes into picking hardware for local models.

The card is never the expensive part. The hours you spend keeping it happy are.

Stop tuning once the workload clears its latency and quality bar with real headroom left, not zero. That margin absorbs a longer document, one more concurrent user, a hot afternoon that throttles the card, or the next runtime release nobody announced. What I don't have is a clean way to price that risk before I've paid it once. You can budget hours for a stack you understand. You cannot budget for the kernel that doesn't exist yet, the one that shows up when next quarter's model needs an attention variant neither vendor shipped. No spreadsheet cell for that. I just leave more headroom than feels necessary, and hope I guessed right.