Hardware for a local voice pipeline

And that's usually where the money gets spent wrong: on the piece everybody can name, while the piece nobody names is what the person on the other end actually waits on.

Four jobs, one queue, different appetites

A voice pipeline runs as four jobs stacked behind each other, recognition, diarization, generation, synthesis, and each leans on the hardware differently. Recognition wants a short compute burst the instant audio stops, then idles. Diarization cares more about memory bandwidth than raw flops. The language model wants VRAM held resident between turns so it isn't reloading weights on every reply. Synthesis just wants a free CPU core. Spec around any one of these and the other three quietly tax the latency you already thought you'd paid for. Run a local Whisper pipeline as its own service and this is the stage that surprises people, never slow alone, only when three other jobs want the same core at once. Sizing rules from a piece on hardware for local LLMs don't transfer here, that's one appetite, this is four.

Write down what the box actually does

Before changing anything, run the pipeline once as a person would, and record it: model, runtime, launch flags, cold versus warm. Time to first token covers loading and prompt work, decode rate covers generation, turn time is what the user feels. For a background job, swap that last figure for completed jobs per hour. I keep this as a plain text file, nothing cleverer:

model + runtime + launch flags
fixed input set (not random)
cold p50 / warm p50 / p95
peak RAM+VRAM, wall power if relevant
failures and abstentions, not just speed
change made, reason, date to recheck

Skip that last line and the file becomes trivia, you'll rerun this in three months with no memory of why the numbers looked the way they did.

The failure that never throws an error (voiceai)

The case that costs you is quiet. A model partially offloads to CPU and still answers, just slower. A cache misses and nothing logs it. A disconnected client's request sits in a queue, holding a GPU slot nobody's watching. A fallback path silently reroutes audio, and the transcript still looks fine. None of that throws an error. Watch OS counters alongside the app's own timer, and change one variable per run, not three. Run each test more than once. A good number next to one you never rechecked isn't evidence, it's a guess with a decimal point.

Assign hardware by stage, not by model name

What's held up for me: optimize the whole turn, assign hardware stage by stage, instead of just buying the biggest card and hoping. It's a duller shopping list, but a slowdown then points at one stage, not the whole machine. Stop tuning once the workload clears its target with some room to spare, that margin is what absorbs a longer recording or a second user later. I still haven't found a clean way to catch a silent fallback before a user does, and I'm not sure a log line ever fully will.