Logging local inference: you still need receipts
When I moved most of my day-to-day inference off cloud APIs and onto the 3090 box in the closet, I had a tidy list of what I was gaining. It took about a month to notice what I'd quietly lost: the dashboard. Every hosted provider ships one: requests, latencies, error rates, spend by key. Ollama answers on :11434 and remembers nothing.
Local inference is unmetered by default, and unmetered drifts into unexamined.
the argument I lost by default
Four of us point tools at that box (code-review scripts, a ticket-triage bot, two editors). One Thursday in late May, a colleague said the box felt slower. I said placebo. Neither of us had a single number, so the discussion went where data-free discussions go: nowhere, twice a week, for two weeks.
He was right, it turned out. I only know that because of what I built next.
the grafana detour
My first fix overshot. I spent most of a weekend standing up Prometheus, Grafana, an exporter, dashboards with gauge panels in tasteful green. It ran for twelve days before I tore it all down: the monitoring stack had more moving parts than the thing it monitored, and nobody had opened Grafana since the Tuesday I shared the link.
What replaced it is a 118-line Python proxy on :11435 that forwards to Ollama on :11434 and appends one JSON line per request to a file. That's the entire system. Ollama already does the hard part: every response carries prompt_eval_count, eval_count and eval_duration, so token counts and tokens-per-second come free, and the proxy just divides and writes. The deeper measurement rabbit hole is its own topic (see measuring local tokens).
{"ts":"2026-05-26T14:03:11","model":"qwen3-coder","caller":"ci-triage",
"prompt_tok":1834,"out_tok":412,"tok_s":26.4,"ms":15602,"status":200}what the file caught
Tokens per second is a hardware health metric wearing a software costume.
Once I could plot tok/s per day, the placebo turned out to be a slope. A steady 33 to 34 through April. Then 31. Then 29. By the last week of May, 26 and change. Same model, same quant, same mix of prompts. Roughly a 22 percent slide, slow enough that every individual day felt normal.
nvidia-smi showed the card sitting at 84°C and throttling. The cause was exactly as glamorous as you'd expect for a computer that lives in a closet: the intake filter had felted over with dust. Ten minutes with a vacuum cleaner, and the next morning's median read 33.8.
The same file caught a second thing I'd never have seen: a retry loop in one of my own scripts, re-sending an identical 22k-token prompt a few hundred times overnight. No error, no crash. Just heat, with my name on the caller tag.
what I log, and what I deliberately don't
Each line gets a timestamp, model, latency, token counts, tok/s, status, and a caller tag. Every tool sends an X-Caller header, so blame is cheap. Prompts go to a separate archive: first 2,000 characters only, deleted after seven days. That archive exists for debugging bad answers, everyone on the team knows it's there, and it has paid for itself three times. Responses I don't keep at all; when I need one, I replay the archived prompt, which doubles as a cheap regression check.
the platform urge, resisted
My actual position: this should stay a boring little script. The pull toward platformhood is constant: alerts, retention tiers, a proper web UI. And for a production agent fleet I do want the real thing, which is a different article entirely (agent observability). For one box and four people, a JSONL file plus a weekly cron that posts median tok/s to Slack is honest infrastructure.
The limitation, stated plainly: a flat file doesn't page anyone. I found the dust because I happened to plot that week, not because anything alerted. The day the box serves a whole team is the day this grows thresholds and an on-call habit — at which point you're running a proper shared Ollama server and should treat it like one.
Cloud gives you receipts whether you want them or not. Local makes you print your own. Mine cost 118 lines and found the dust.