A portfolio of small models can beat one large resident model

A 24 GB box serving embeddings, extraction, coding, and general chat off one resident model is a capacity problem before it's a modeling one. Load the biggest model that fits and hope it covers everything, and you get something mediocre at each job, sitting on VRAM the embedding job never touched. Before touching a launch flag, or routing across specialists via building-an-autorouter, decide what number you're moving: first-token latency, jobs per hour, resident model count, energy per task, fewer reviewer corrections.

Pin it down before you split it up

A specialist portfolio beats one resident model if you can prove it, which takes a baseline that doesn't wobble. Pin the model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings. Test on real inputs from the workload, including awkward cases. Run once cold, model unloaded, storage untouched: the path a real user hits first. Then run warm, repeatedly, until cache effects, queue backlog, and thermal drift show themselves. A config that wins short and warm can lose once prompts get long.

Watch the phase, not the headline number

Total time is a bad summary. Break every run into phases:

• queue wait
• model load or activation
• prefill to first token
• decode rate and completion
• peak RAM, VRAM, power, and swap
• quality gate: pass, retry, abstain, or repair

Raw tokens per second is diagnostic, not the product: automation counts records per hour, coding counts review and correction time, chat counts p50/p95 first-token latency at realistic conversation lengths. Change one variable per test and keep the outputs, not just the metrics: quantization and context compression can shave time while truncating an answer. The quieter failure is a shelf of near-identical specialists whose reloading erases the gain, and local runtimes hide it well, paging memory, offloading layers, missing a cache, or falling back to a generic kernel until someone checks a log.

Write the reason down, then leave room

Keep a specialist when it moves quality or resource use by an amount you'd defend out loud. Write the conclusion into the result file with the workload, the date, and the reason, plus what should trigger a retest: a new model family, a driver update, longer contexts, a different traffic mix, or the numbers turn into folklore nobody can re-derive. A five-percent win from an undocumented patch or manual warming is a bad trade unless you're on call. Leave headroom past the target: spare memory for prompt variance, spare queue slots so interactive traffic skips the batch queue, thermal margin for sustained load. Boring configs age well.

I still don't have a good answer for what forces anyone to reopen that result file. It's a comment in a directory, trusted to whoever inherits the box, and I haven't found a way to automate the check without building something heavier than the portfolio it protects.