keep_alive and the cold-start tax
The complaint that finally made me look at this wasn't about quality. A colleague tried the local setup I'd been evangelizing, asked one question, got a genuinely good answer, came back after a meeting, and sat through a 12-second pause before the first token of the second one.
His verdict was that it's slow, which was wrong in every technical sense and right in the only sense that matters.
By the time he asked that second question, the model was already gone: unloaded by Ollama's five-idle-minute default. Intermittent use is most real use, and it pays the full disk-to-VRAM load on nearly every question. That's the cold-start tax, and no spec sheet mentions it.
three values cover everything
keep_alive takes a duration per request, or OLLAMA_KEEP_ALIVE sets the server default. The stock 5m suits a box that juggles many models. -1 pins a model in memory until you say otherwise. 0 unloads it immediately, which sounds pointless until you need to evict something right now to make room for a bigger model. And you can warm a model without prompting it:
curl localhost:11434/api/generate -d '{"model":"qwen2.5-coder:7b","keep_alive":-1}'An empty generate loads the model and, with that flag, pins it. That line runs from a login script on the 3090 box, so my daily model is warm before I've opened a terminal.
Nobody benchmarks the twelve seconds before the first token, yet it's the only number people's hands remember.
my split, and what it costs
Pinning has a price: a pinned model is VRAM spent whether you're using it or not. My split is one of each. The 7B coder quant, around 5 GB, stays pinned because I hit it dozens of times a day; the 32B keeps the default five minutes because I use it in bursts, so the first question of a burst pays the load and the rest ride warm. On 24 GB that budget barely closes. The offload math is worth doing on paper before you pin anything.
the four days I ran on CPU
In mid-June I got greedy and pinned both. Together they don't fit in 24 GB, and Ollama did what it always does when VRAM runs short: split the 32B across GPU and CPU without a word of complaint. Generation dropped to roughly 5 tokens a second: clearly worse, still working, exactly the failure mode that survives. I blamed the quant, then a driver update. On day four I finally ran ollama ps and there it was, 41%/59% CPU/GPU.
Unpinning the big model fixed it. Took one second.
ollama ps is a reflex now, the way git status is. If you're juggling more than two models on one card, that's a discipline of its own.
Pin what you touch hourly; leave the default on everything else. It's one line of tuning against the worst first impression local inference makes.