Temperature isn't a vibe: sampler settings that matter locally

the week the keys mutated

In early June I had a small extraction pipeline on the Mac Studio turning messy supplier emails into JSON (an 8B model, nothing exotic, a few thousand documents a day). On Wednesday someone downstream pinged me: the parser was throwing on roughly one document in twelve. The model had started renaming keys mid-output. The first thirty objects would say invoice_id, then somewhere deep in a long array it would switch to invoice_identifier, then invoiceId, like it had gotten bored of the word.

I blamed the model. Drafted the commit that swapped it for a bigger one, wrote a little rant in the commit message and everything. Before pushing, I diffed my request settings against a colleague's setup that didn't misbehave, and the only difference was repeat_penalty: mine 1.1, his 1.0. That default (which I never chose) taxes tokens for having appeared recently. In prose, that's what stops loops. In a long JSON array the keys are supposed to repeat, and by object forty the penalty had made invoice_id expensive enough that the sampler reached for a synonym.

The commit never shipped. The rant got deleted, with mild embarrassment.

five knobs, one honest sentence each

Local frontends expose a pile of sampler settings, and most write-ups treat them like folklore. The short version I actually believe:

• temperature: how flat the next-token distribution gets; low sharpens toward the single likeliest token, high flattens toward chaos.
• top_k: keep only the k likeliest candidates; a blunt cap that mostly matters at high temperature.
• top_p: keep the smallest set of candidates whose probabilities sum to p; adaptive in theory, clumsy at the tails.
• min_p: keep candidates at least min_p times as likely as the best one; a relative floor that scales with the model's confidence.
• repeat_penalty: retroactively taxes recently used tokens; medicine for loops, poison for structured output.

min_p is the one nobody sets

A fixed top_p keeps too much junk when the model is unsure and cuts too deep when it's confident; a relative floor sidesteps both failure modes. My default for anything creative is temperature around 0.9 with min_p at 0.05 and top_p effectively off. You get variety without the occasional word salad. The catch: it isn't universal. Ollama's native API honors min_p, but plenty of OpenAI-compatible layers silently drop it on the floor, so confirm what your server actually received before concluding it does nothing.

Half the local-model complaints I've debugged this year were sampler defaults wearing a model costume.

my presets, per task

Code edits get temperature 0.2 and repeat_penalty 1.0. Extraction gets temperature 0, repeat_penalty 1.0, and a schema. After the invoice incident I stopped asking nicely for JSON and started constraining it, which makes half the sampler debate moot. Brainstorming and naming get 0.9 with min_p. Long summaries sit around 0.4. None of this is sacred. All of it is written down in one options file per task, which is the part that actually matters.

Presets in a file beat vibes in a UI.

check the defaults before you blame the model

Models ship with different Modelfile defaults. One pulls with temperature 0.6 and top_p 0.95 baked in, another inherits the runtime default of 0.8, and neither will tell you unless you ask:

ollama show mistral-small --parameters

Every time I A/B two models without pinning the sampler first, I'm really benchmarking two sets of defaults against each other and calling it a model comparison. My model probation routine pins temperature, min_p and repeat_penalty before any new model gets judged, and that one habit has quietly resolved more model complaints than any actual model swap I made this spring.

Temperature is a per-task decision, not a setting you pick once and forget. When the output goes weird, check the sampler before you check the leaderboard.