Ollama in Docker: three gotchas and a compose file

I resisted containerizing Ollama for a long time, on the grounds that the installer is one command and Docker mostly adds new ways to be silently wrong. What changed my mind, back in April, was needing the identical setup in three places: my desk machine, the office box that became our team server, and a CI runner. Three hand-configured installs drift apart within a month. One compose file doesn't.

Here's the file, then the three ways it bit me before it settled down.

services:
  ollama:
    image: ollama/ollama
    gpus: all
    ports: ["11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    healthcheck: {test: ["CMD", "ollama", "ps"], interval: 30s}
volumes: {ollama: {}}

gotcha one: the GPU that isn't there

Ollama in a container without GPU access doesn't error. It runs on CPU (at somewhere between a fifth and a tenth of normal speed, depending on the model), and every endpoint answers, every test passes, and anyone who tries it concludes local models are garbage. GPU access needs two halves: the nvidia-container-toolkit installed on the host, and the gpus line in the compose file. Forget either half and you get the slow path with zero complaints. On a fresh GPU box in May I had the compose right and the toolkit missing. It took me a day and a half to notice.

The first command after any deploy is now docker exec ollama ollama ps, where I read the processor column rather than the logs.

gotcha two: the volume, or a terabyte of re-pulls

Models live under /root/.ollama inside the container. If that path isn't a named volume or a bind mount, every container recreate starts from an empty cache. Recreates happen more often than you'd think, with every image update or any CI job that runs docker compose down.

This one got me properly. Our integration suite pulled a roughly 13 GB model at the start of every CI run, on each merge to main, nine or ten times a day, because the runner's containers were ephemeral and nobody had mounted the model cache. It ran that way for a little over a week before the office connection started feeling sticky and I pulled up the router's bandwidth graph: just shy of a terabyte of repeated model pulls. The fix was one persistent volume on the runner, and the suite got about six minutes faster as a bonus.

Nothing in CI is more durable than a mistake that still passes.

gotcha three: healthy is not warm

The obvious healthcheck (does 11434 answer) goes green the moment the server starts, long before any model is in memory, so the first real request eats a multi-gigabyte load while your orchestrator swears everything is fine. My compose uses ollama ps as the check because the binary is already in the image, which ships without curl, at least the last time I looked. Where first-token latency actually matters, I add an init step that fires a small generate request with keep_alive set, so the model is warm before traffic shows up.

when bare metal wins

On a Mac, skip all of this. Docker on macOS runs inside a VM with no Metal passthrough, which means containerized Ollama on Apple silicon is CPU-only: the one gotcha no flag can fix. My M2 Ultra runs Ollama straight from the installer and always will. The container story is a Linux-with-NVIDIA story.

For the shared box and CI, it was worth it: the reproducibility is real, and treating model infrastructure like any other service we ship pays off in habits elsewhere. For the machine you personally sit at, I honestly wouldn't bother. The installer is fine. Save the YAML for computers other people depend on.