The minimum observability for local inference
You don't need a monitoring team to know why a local model feels slow. You need five timestamps, captured consistently, and the discipline to write down what you did about it. That part is what everyone skips.
Local AI talk gravitates to a model name or a peak throughput number. The better question is what's waiting behind the API: a private chat twice an hour, a coding loop firing small completions all day, a nightly batch job, sharing one box and needing opposite handling. The stall a person notices mid-sentence has nothing to do with the throughput a batch job cares about at 3 a.m.
What I write down before touching a knob
Before reconfiguring anything, I capture a baseline: one named model, one fixed prompt set, the exact launch command, the artifact hash, the prompt template. Skip the last two and the comparison stops meaning anything once you've forgotten which build you tested.
I track the request itself, not one duration figure: time it arrives, clears the queue, model-ready, first token, completion, plus model identity and memory state. Time to first token is loading and prompt work, steady rate is decoding, completion time is what the user feels, and it can look fine even when everything upstream was a mess. Batch work: valid jobs per hour. Interactive work: the slow waits someone remembers.
The record stays plain: artifact, runtime, launch flags; workload and fixed input; cold- and warm-start times against p50 and p95; peak memory and wall energy; quality failures and abstentions; a decision, an owner, a retest date. A benchmark without a decision is trivia, and a setting with nobody's name on it is folklore.
The failures that never throw an error
The shortcut is logging total duration and guessing whether the delay was load, prompt, queue, or decode. It survives because the output looks plausible. Local inference fails quietly: a model partially offloads to CPU, a cache stops hitting, swap creeps up, a queue holds requests whose client already hung up, a fallback path changes where the data goes. None of that throws an exception.
I watch runtime logs and OS counters while the test runs, one variable at a time unless I'm comparing whole configurations, repeating enough to tell a durable gain from luck. I read the outputs, not just the speed: a quantized model that answers faster and gets more wrong isn't an improvement, and a latency win that costs accuracy gets written down beside the gain.
Operational simplicity matters too. A clever fix needing manual repair after every driver or model update isn't free. I'd rather run something boring and reproducible from a service file or a short script than something fast only one person can rebuild. Raw logs stay out of the report; the hashes and commands that made them stay in it.
Where the margin actually goes (observability)
The rule that's held up for me: collect phase-level metadata, keep the user's text out of routine telemetry, full stop. That second half isn't optional with anything private in the box, and it's nearly free: none of the five timestamps need prompt content. Visible limits get routed around, scheduled, or priced in; invisible ones show up as an outage nobody predicted.
Stop tuning once the workload clears its quality and latency target with real headroom left. That spare margin isn't waste: it absorbs a longer document, an extra user, a warm afternoon, or whatever the next runtime release changes. Efficient local inference is the discipline of spending capacity only where it changes what the user gets back.
What I still don't have a clean answer for: once two or three workloads share one accelerator, the per-request timestamps I want and the OS-level counters I have live at different layers, with no shared clock. I can tell you the GPU was saturated. I can't tell you which job paid for it.