Why memory bandwidth predicts local token speed

So no, the flops number on the spec sheet won't save you here. What actually throttles decode-time throughput on a single request is that the accelerator has to haul a big chunk of the model's weights out of memory for every token it prints, unable to batch. Compute-bound benchmarks describe a different world: batches queued, cores saturated. A single chat session at batch size one is not that world. It's bandwidth-bound. Full stop. Treating it like a compute problem is how people end up buying the wrong card.

And a real machine rarely runs one workload anyway. A private assistant fired off twice an hour, a coding loop running nonstop, and a nightly extraction job can share a box and still want different things from it: a short wait before the first word, a decode rate that holds steady, or just finished work per hour.

what the stopwatch is actually timing

Before you swap hardware or start on quantization, pin down a baseline: one named model, one fixed prompt set, the exact server command, written down. Tedious, but it's the only way a second run means what the first one meant. Time to first token is mostly loading and prompt processing. Steady per-token rate once generation is underway is decoding, where bandwidth shows up directly. Completion time is what a person or a downstream job feels. I keep this as a plain text file, not a dashboard, because a dashboard tempts you to stop reading the numbers.

model + quant + launch flags
prompt set (fixed, versioned)
ttft p50/p95, decode tok/s p50/p95
peak vram, queue depth
decision + retest date

A row with no decision next to it is trivia by Tuesday.

buy the bandwidth, not the flops

The failure mode I keep running into is someone using a gaming or general compute score to predict how a single chat session will feel, then confused when the "faster" card underperforms an older one with wider memory. It survives because the model still answers, just slower, and slow is quiet. Some of that slowness hides in soft failures that never throw an error: layers offloaded to CPU, a KV cache that stopped fitting, a queue holding a client that already disconnected. None of it shows unless you watch runtime logs and memory counters while it runs.

Here's the rule I'd carry into a purchase: treat bandwidth per dollar and capacity per dollar as two separate line items, not one "how big a model fits" question, and check the offload math before assuming a partial fit is free. Less flattering way to shop. Nobody brags about GB/s. But it gives you a stack whose limits you can see coming: a visible one gets scheduled around, an invisible one turns into a late-night page. Stop optimizing once the workload clears its bar with headroom to spare, because that headroom absorbs the next runtime update or a document far longer than usual.

None of this notebook discipline replaces running your own model on your own prompts, and half the time I set the baseline file up once and don't reopen it until something breaks.