New model dropped. Now what?
the every-other-thursday problem
A new local model drops, the timeline melts, and by lunch someone in our team chat has declared the incumbent obsolete. This happens, on average, every other Thursday. I used to pull each one onto the Mac Studio the same evening and form an opinion by midnight, and those opinions were garbage, anchored on whichever two prompts I happened to try and whatever mood the sampler defaults put the thing in. Model churn is the weather here. The fix had to be a routine, not a resolution.
the gauntlet
Every candidate now faces the same 20 prompts before I'm allowed an opinion. They're pulled from real past work, ugly ones included:
- six code edits from actual client repos, including one where the right move is refusing to touch adjacent code
- four extractions into strict JSON, one with a deliberately broken input
- three SQL rewrites, plus a regex I once got wrong myself in 2024
- two tool-calling transcripts replayed step by step
- a long-context needle, a Czech-to-English translation, one refusal-bait prompt, and one where the correct output is a clarifying question
Sampler settings get pinned first, or you're comparing defaults, not models. Outputs land in files named a and b, and I grade before looking up which is which. Blinding this cheap feels silly and works. Twenty prompts is not an eval suite; when scores come out close I widen the set, and for anything that will run unattended I borrow the heavier machinery from evals and LLM-as-judge. For a Thursday verdict, twenty is plenty.
disk is part of the discipline
In April, ollama list confessed to 21 models and 348 GB, most of them one-evening flings. Now losers get removed the same week (ollama rm, no ceremony) and anything that hasn't been the right answer to a real task in six weeks loses its slot.
A model I can't name a job for is disk space with a leaderboard score.
The pull is free. The hoarding isn't.
probation, and the time I cheated
A gauntlet winner still doesn't touch client work for two weeks. It runs the boring lane first (commit messages, summaries, extraction) right next to the incumbent, and it gets promoted only if nobody downstream noticed anything odd. That rule exists because of March: my old favorite coder lost to its successor 14 to 6 on my own gauntlet, and I kept routing edits to it for another month anyway. I knew its quirks. I liked its diffs. I had eighteen saved presets tuned to it. Sentiment, with extra steps.
The successor was better, my own sheet said so on day one, and I shipped a month of slightly worse work anyway. Trust the sheet you built for exactly the moment you'd rather not.