Use different sampling settings for different jobs

A 12 GB card serving a chat session at nine in the evening is the same card running the nightly extraction batch at three in the morning, and it does not know the difference between the two jobs unless something tells it to behave differently.

Most people never tell it. One temperature and one top-p, set once, ride along into everything after: fixing a function, pulling a field from a ticket, riffing on copy at midnight. That's a mistake, and a boring one.

Temperature doesn't backfill a thin prompt

The constraint worth pinning down first: decoding settings control repeatability and diversity, nothing more. Whatever sampler you run, temperature, top-p, top-k, it only reshapes which plausible token gets picked; none of it puts evidence into a context that never had it. Miss the stack trace on a repair task and no sampler tuning rescues that, the answer just comes out wrong in a different shape. That's why repair and extraction want the opposite of brainstorming: same input, same output, run after run, so a regression reads as a regression, not noise, which is the case for running automated code review at near-zero temperature rather than as a creative task. Brainstorming wants the sampler loose, since the point is surfacing options you wouldn't write yourself. Prose sits closer to repair than most assume: it shouldn't read differently on every regeneration.

Write the baseline down before you touch anything

Before touching a setting, write down what you're running: one named model, one fixed prompt set, the server command with flags, the model artifact, the prompt template, since all of it drifts and can invalidate the comparison. Define presets per task, hold prompts constant, and run each one enough times to see a success rate and its variance, not one lucky pass. Track the request end to end: time to first token for loading and prompt work, steady token rate for decoding, completion time for what the downstream pipeline actually gets. Add peak memory, queue delay, and wall power when they change the decision. Count completed, valid jobs per hour for automation; count the slow waits people remember for anything interactive, a close cousin to the setup behind running evals with an LLM judge. I keep the record short, one line per field:

• model artifact, runtime, launch flags
• workload and the fixed input set
• cold start, warm start, p50, p95
• peak memory and wall energy draw
• quality failures and outright abstentions
• the decision, its owner, the retest date

That last line is the one people skip, and it matters most: a benchmark with no decision attached is trivia, and a setting nobody owns turns into folklore.

The mistake that still looks like it's working

The tempting move is retuning temperature every time an answer looks off, treating a bad output as a reason to nudge the sampler instead of a symptom of something else. It survives because the output stays plausible either way, so retuning feels like progress when it's just noise. Local inference carries soft failures that never throw an error: a model offloads part of itself to CPU because it no longer fits in VRAM, a cache misses unnoticed, swap creeps up, a queue holds requests from clients already gone, or a fallback path changes which machine, and which privacy boundary, answers the request. Watch logs and OS counters while the test runs, change one variable at a time unless comparing whole configurations, repeat enough to trust the result, and read the outputs rather than treat speed as proof two setups behave the same. If a change makes something important worse, note that cost beside the gain.

Boring configurations beat clever ones on a Tuesday

Operational simplicity deserves its own column, and most comparisons skip it. A clever optimization needing manual repair after every driver update or model swap is not free, whatever the benchmark says. I'd rather run something boring that a service file or a short script reproduces exactly than chase a setup only one person can revive. Keep the hashes and exact commands, not the raw artifacts, so a rerun means the same thing later. The rule worth carrying forward: conservative, close-to-deterministic presets for automation, diversity spent only where it buys something real. In practice that means the coding loop and the extraction job stay tight while brainstorming gets to wander. Less exciting than picking the biggest model or newest card, but it leaves a stack with visible limits: schedule around them or price them, instead of a Friday-afternoon emergency nobody budgeted for.

Stop when the headroom is doing something

Stop tuning once the workload clears its quality and latency target with margin left over. That margin covers the message that runs long, the extra person logging in at the same moment, or next month's runtime update shifting behavior underneath you. Efficient use of a local model is mostly spending its capacity only where that changes the outcome, and leaving the rest alone. Next thing to check: pull last night's extraction log and count how many completions came back malformed at whatever preset is live now, because that number, not a vibe about the model, says whether the retest date is overdue.