Three models, one GPU: the juggling act

Since February the closet box has been trying to be three servers at once: an embedder that every search feature I own depends on, a small chat model my wife and my scripts share, and a 32B coder that eats the card nearly whole. One used 3090, 24 GB. That sounds roomy until you do the arithmetic.

the split that survived contact

The current residents: nomic-embed-text at about 550 MB, an 8B chat model around 5.6 GB in Q4, and a 32B coder at roughly 19 GB for the weights, plus KV cache, which at 16k context wants another 2 GB and change. Embedder plus coder comes to about 21.7 GB and fits with a little headroom. Embedder plus chat is a trivial 6.2 GB. All three together want around 27 GB, and that card does not exist in my flat. So the game is choosing which two live on the card at any given moment.

The policy that survived five months: the embedder is pinned with keep_alive set to -1, because a cold embedder breaks search everywhere and its 550 MB is the cheapest rent I pay. The chat model gets 30 minutes. The coder gets 10, with OLLAMA_MAX_LOADED_MODELS set to 2 so Ollama evicts the chat model rather than trying to squeeze. Per-model keep_alive does the actual scheduling; I just set the incentives.

pinning versus swapping is a latency budget

Loading 19 GB of coder weights off NVMe takes 7 to 8 seconds on my box. Whether that matters depends entirely on the shape of the day. During interactive coding it's murder: an 8-second stall on the first request after every quiet spell reads as broken. So during work hours I flip the policy: coder pinned, chat evictable, and my wife's requests eat the cold start instead, which she notices roughly never, because a chat model reloads in about 2 seconds. Overnight, the cron agents don't care about anything, so everything goes evictable and the card breathes. The offload math is worth internalizing once; after that it's just budgeting.

Pinning everything by default is hoarding, and a hoarded card serves one workload badly instead of three workloads well.

the Tuesday everything spilled and nothing said a word

Mid-June, a Tuesday night, I raised the coder's num_ctx from 16k to 32k so an agent could chew a whole test file in one pass. KV cache grows with context; the extra gigabytes pushed the total past what the card had free with the embedder pinned. Ollama didn't refuse. It quietly placed a slice of layers on the CPU and carried on. Generation dropped from about 27 tok/s to about 4, every request still returned fine, and the overnight run finished green after doing maybe a seventh of the planned work. I found it on Thursday, from a log line, by luck.

The worst failure mode on a shared GPU is everything still succeeding at one seventh of the speed.

how I catch it now

Two habits. First, ollama ps after any config change, eyes locked on the PROCESSOR column:

$ ollama ps
NAME               SIZE    PROCESSOR         UNTIL
my-coder:32b       23 GB   22%/78% CPU/GPU   9 minutes from now

Anything other than 100% GPU in that column is the alarm going off, whatever the logs claim. Second, I log per-request throughput and alert when the coder dips under 15 tok/s three requests in a row (thermal trouble, spills, and my own config mistakes all trip the same wire, which is the point of the wire).

I also stopped flying at the ceiling. The coder now runs at 24k context instead of 32k, keeping about 1.7 GB free, because model updates have a habit of arriving a few hundred megabytes heavier and I'd rather donate a little context than replay that Tuesday.

The box carrying all this is the same one I wrote up in a year with the closet server. The juggling act is the price of refusing to buy a second GPU. So far it's a price I keep choosing to pay.