Model churn: my quarterly ritual of re-testing local models

The July run is this Saturday. Twenty-three prompts, four local models, one evening, and if history is any guide, the result will be "change nothing". I do it anyway, and the reason is an argument I lost last October.

the argument I lost

We were picking the default local coding model for the team, and I defended the incumbent, a 14B I'd been running since summer and had tuned my prompts around. My case was familiarity dressed up as evidence. A teammate did the annoying, correct thing: ran the incumbent and the newer release against twelve real tasks from our backlog and posted the outputs side by side. The new model was plainly better on nine of the twelve.

What stung wasn't being wrong; it was recognizing the mechanism. I had months of prompt-tuning sunk into that model, and somewhere along the way the investment had quietly turned into loyalty. Sunk cost with a chat interface.

So now I re-run the test before I voice the opinion.

twenty-three prompts, frozen

The eval file is a jsonl I started with twenty prompts and have appended three to since: append-only, never edited, because editing the test to flatter a model defeats the point. Everything in it is lifted from real work:

• SQL generation against a real schema, including one deliberately nasty join
• a Terraform diff review with one planted bug
• commit-message summarization from actual diffs
• one regex to explain, and one subtly wrong regex to catch
• two refusal-bait prompts to flag over-cautious models

Scoring is unglamorous. About half the answers are checkable at a glance; the fuzzy ones go to a judge model with a rubric, which I trust for ranking candidates and never for absolute scores. evals-and-llm-as-judge is where I worked out that boundary. A full run across four candidates takes about forty minutes on the 3090 box, and reading the outputs takes the rest of the evening. Deciding what's even worth downloading between quarters is its own filter, or the ritual would eat every weekend.

the deleting part

In January, ollama list reported 287 GB across thirty-one tags. Quants I'd tried once. An abandoned vision experiment. Three sizes of the same family, kept out of vague completionism.

Now the rule is mechanical: any model not run in ninety days gets removed on eval weekend, no appeals. The disk is the visible cost. The invisible one is that every installed model is a standing invitation to procrastinate by benchmarking instead of working. I keep one champion per role (coder, general-purpose, embedder) and my current picks only change when the file says so.

Loyalty to a model is just an eval you're refusing to run.

most upgrades don't matter, and that's the finding

Three of my last four quarterly runs changed nothing. The new releases were genuinely better on paper and indistinguishable on my twenty-three prompts, because my prompts are bounded tasks: classify, summarize, translate to SQL, review a diff. Bounded tasks saturate early. The visible deltas live in long agentic chains, and I route those to frontier models anyway.

That "nothing changed" result is the product, though. It's ninety days of permission to ignore release-day noise, bought for one evening. And when a run does flip the ranking (once so far), I switch without ceremony, because the file has more credibility than my attachment does.

Known limitation, stated plainly: twenty-three prompts overfit to me. They can't catch regressions off my beaten path, and they'd embarrass themselves as a public leaderboard. I don't care. When my file disagrees with a public benchmark, I go with the file, because I ship my tasks, not the benchmark's.

Saturday, then. My money's on "keep everything", and I'll be slightly disappointed if I'm right.