Build the eval set from embarrassing failures

A single workstation running a local model has one GPU, one pool of VRAM, and no autoscaling anything: whatever you throw at it competes with your editor and whatever else you forgot was still open, for the same fixed pile of memory.

That's the box most of us run inference on, not a rack with headroom to spare. My running example is a local assistant that swaps models and prompt versions every few weeks, the kind of churn anyone who's lived through Ollama's model shuffle recognizes on sight. Nothing about it is exotic: ordinary hardware, an ordinary job, no lab conditions, no claim my numbers generalize to your machine. What I insist on is writing the job down before touching a setting: chat, code completion, document extraction, an overnight batch. Decide what a good result looks like before you go looking for one.

Steal the eval cases from your own screwups

The method is unglamorous. Every time the assistant gets something wrong in a way that costs me real time, that case goes into the set: the input, the expected answer, enough context to reproduce it, sensitive bits stripped first. Rerun the whole set on every change that matters, same inputs every time, no quietly swapping in an easier seed. Save the launch command next to the result, because in six weeks you won't remember which quant or sampler settings produced that number.

One warm run tells you almost nothing. Include a cold start if a human will hit one, run long enough for throttling or queueing to show, and keep the output text next to the timing, not just pass or fail. A setup that answers twice as fast but gets the wrong answer isn't a faster version of the old one, it's a different system wearing the same name tag.

The fields I refuse to skip

Only a few numbers actually move the decision:

• time to first token
• tokens per second once generation is running
• peak memory during the run
• wall power draw, when you can measure it
• task success against the expected answer

Medians hide the run that took four times as long because something grabbed the GPU for a second, exactly the run that annoys whoever's waiting. Track a slow percentile too, because it catches what the median erases. Note the friction as well: a prompt that needs retyping, a model that needs a manual restart. That friction is often the real reason a setup gets abandoned.

What actually goes in the result file

I keep the record as plain text, one block per run, so grep still works once I've forgotten the tool:

task:      named, single workload
inputs:    fixed, versioned, redacted
verdict:   pass / fail / abstain
latency:   cold, warm, p95
resources: memory, power, disk
call:      keep, revert, retest

Nothing here is clever. A proper framework holds more, but this gets filled in after a change at eleven at night, and a format you'll use beats one you'll admire once.

'It loaded' is not a result

The trap I keep falling into is building a set out of easy cases: they pass, produce a reassuring number, and never catch a real regression. "It loaded" is not a performance result. "The answer looked fine" is not an evaluation, it's a vibe. Trust the runtime logs and the OS counters over whatever flag you asked the process to respect, because plenty of configurations that technically run are quietly ignoring the setting you think you turned on.

When two runs disagree, change one variable and write down the mechanism you expect before rerunning it, or the benchmark degrades into a folder of anecdotes with numbers attached. There's a maintenance cost too: a fragile five-percent win evaporates the next time a driver, a model, or the runtime gets updated, and anyone who's sat through a real model migration knows how often "compatible" means "produces subtly different answers." A small script, a dozen cases, and a plain-text file are enough to retest after every upgrade, no observability platform required for one workstation.

Where I stop tuning

The rule for what stays in the suite comes from production pain, not curiosity: a failure mode that never bit anyone doesn't earn a slot. Test the smallest plausible change first. Keep headroom in reserve rather than spend every gigabyte chasing a number, because that headroom absorbs a longer prompt, a stray background app, or next month's model. Once the workflow hits its latency and quality target, stop, not because it couldn't go faster with more tuning, but because tuning past that point is a hobby.

Local models reward exactly this restraint. Hardware matters, but clear workload boundaries and a handful of controlled measurements get you the first real gain. The setup worth keeping isn't the one with the flashiest number, it's the one whose failure modes you already know by name.

None of this scales past one person's workstation, though. The moment more than one of you touches the same box, or the set outgrows what fits in your head, this plain-text approach starts creaking. I still haven't fixed that part, and I might just be putting off admitting a spreadsheet was going to win eventually.