One GPU box for the whole team

When our ML contractor wrapped up in March, he left behind a Ryzen workstation with an RTX 3090 that nobody claimed. It sat in a closet for a couple of weeks before I put Ollama on it, changed one environment variable, and told the other four backend devs it existed. That's the entire origin story of our team inference box, and three months in I can tell you fairly precisely what one shared 3090 is good for.

The one variable is OLLAMA_HOST. Ollama binds to localhost by default (the right default, the wrong setting for a shared machine), so you set 0.0.0.0 and the LAN can reach port 11434. The knobs that took longer to get right are about concurrency, and on Linux they live in a systemd override:

Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_NUM_PARALLEL=3"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_MAX_QUEUE=64"

OLLAMA_NUM_PARALLEL is the sharp one. Every parallel slot needs its own context allocation, so raising it multiplies KV-cache memory; on a 24 GB card with a 7B coder model, three slots is where we stopped, because past that the layers started spilling off the GPU. OLLAMA_MAX_LOADED_MODELS at two lets the embedding model stay resident next to the coder instead of evicting it on every switch. Requests beyond the slots queue up, and past the queue cap Ollama refuses outright. That sounds rude. It's exactly right: a bounded queue fails loudly, and an unbounded one fails at lunch.

what five people can actually share

Embeddings were the unambiguous win. Everyone points code-search indexing at the box (nomic-embed-text, batched), and five people embedding at once barely registers because each job is small and the model is tiny. Short completions worked nearly as well: commit messages and quick refactor hints, the sub-ten-second stuff. Solo, the 7B does mid-thirties tokens a second, and under normal team traffic short requests still feel instant.

Long generations are where it collapsed. Two people running extended agentic sessions at once, each expecting a 15-minute stream, halved each other into the teens, and everyone's quick requests sat behind them in the queue. One Thursday in May a colleague's overnight batch job was still going at 10 a.m., and the box was effectively his.

A shared GPU is a shared microwave: completely fine until two people show up with turkeys.

There's no environment variable for that one. We settled it socially: long jobs run after 6 p.m. or on your own hardware, agreed in a two-message Slack thread, and the rule has held for two months.

the intern and the missing lock

Now the embarrassing part. Ollama ships with no authentication of any kind, and I ran the box open on the office LAN for about six weeks on the theory that inside the network, who cares. In early May our intern found it (nmap, curiosity, entirely fair) and did the reasonable thing: pointed a personal research script at the nice free GPU. I noticed on a Friday afternoon when everyone's completions went sluggish and ollama ps showed a 32B model I'd never pulled.

The LAN is not a permission model.

He hadn't broken anything; the box answered, so he used it. The fix took one evening: bind Ollama back to 127.0.0.1, put Caddy in front with basic auth and per-key access logs, hand out credentials properly. The logging half turned out to be the more valuable half: once requests had names on them, the whose-job-is-hogging-the-box conversations got a lot shorter.

Three months in, the box quietly handles embeddings and short completions for five people, long work moved to a schedule, and the whole thing deploys from the same compose file as the rest of our tooling. If your team runs genuinely parallel agent workloads all day, one consumer card won't save you. That's proper homelab territory, or a hosted tier, and pretending otherwise just moves the queue into your teammates' heads.