Temperature zero is not a reproducibility guarantee

So here's the part that gets skipped: temperature zero kills exactly one source of randomness, not all of them.

Kernel choices, batch composition, the model build, and tie-breaking on equal logits are all still live, and none of it shows up in a changelog.

Most local-inference writing starts from a model name or a peak-throughput number, backwards for real deployments. A private chat used a couple times an hour, a coding loop all day, and a nightly batch job can share one box and want different tradeoffs.

the log I actually keep

Before touching a setting I capture a baseline: one named model, one fixed prompt set, the exact server command, artifact and template on record.

Then I pin everything, isolate concurrency, repeat the cases that matter, and score properties, not exact prose. Many evals and LLM-as-judge setups go wrong right here: sameness scored as correctness.

Measurements follow the request end to end: time to first token (loading), steady token rate (decoding), completion time (what the user gets), peak memory, queue delay, wall power, completed jobs per hour for batch work, and the slow waits people remember.

model file, runtime version, launch flags
workload, fixed input set
cold vs warm start, p50, p95
peak memory, wall energy
failures and abstentions, counted
decision, owner, retest date

Skip the last line and a benchmark turns into trivia. Skip the owner and a setting turns into folklore.

the trap that still looks right

The mistake I see most is grading generated text by byte-for-byte equality: it survives review because the output reads like plausible prose, and nobody checks what changed underneath.

Local inference fails in soft ways that don't throw errors: partial CPU offload, a cache miss, swap creep, a queue holding disconnected requests, a fallback path moving where the data goes, which matters more than latency for privacy.

None of that trips an exception.

Watch runtime logs and OS counters as it runs, change one variable at a time unless you're comparing whole configurations, and repeat enough passes to separate a durable improvement from a lucky sample before calling it fact. A speed win that costs an important task should show the cost, not hide it.

Operational simplicity earns a column too: an optimization needing repair after every driver or model update, the kind Ollama's model churn forces on people, isn't free because it used to be. Prefer a boring config you can rebuild from a service file or a short script, and keep hashes and commands after dropping the artifacts.

The rule that's held up: build evaluations that shrug off harmless wording changes and catch behavioral ones. Visible limits get scheduled or priced; invisible ones turn into a scramble. Stop when the workload clears its quality and latency bar with headroom, since that margin absorbs the longer document, the extra user, or the next runtime upgrade you didn't plan for.

What I still don't have is a cheap way to tell, after the fact, whether a divergence came from the kernel, the batching, or the tie-break rule, short of re-running the whole stack one layer at a time.