Ollama in practice: context, GPU control, the API, and when to graduate to vLLM

You followed the fifteen-minute setup and a model answers from inside your editor. That's the demo. Running it well (for real work, maybe for more than just you) is a different set of knobs. None are hard; all of them bite people who skip them. Here's the practical layer on top of the quickstart.

First, the mental model. Ollama is llama.cpp with a model manager and an OpenAI-compatible server bolted on. That framing tells you its sweet spot (one box, one-to-a-few users, zero config) and its ceiling (it is not a high-throughput serving stack, more on that at the end). Everything below is about living inside that sweet spot deliberately instead of by accident.

The context-length gotcha that silently truncates

The single most common mistake. Ollama defaults to a 4K context window regardless of what the model supports. Pull a model that advertises 128K and feed it a big file. Ollama quietly drops everything past 4K, and you get confidently wrong answers with no error.

Set it explicitly. Per request via the API:

curl localhost:11434/v1/chat/completions -d '{
  "model": "qwen3-coder",
  "messages": [{"role":"user","content":"..."}],
  "options": {"num_ctx": 32768}
}'

Or bake it into a model (see Modelfiles below). Bigger num_ctx costs VRAM: that's the KV cache the cost calculator sizes. The calculator's ctx/request input is exactly this number; keep them in sync and the VRAM it predicts matches what Ollama actually allocates.

Keep big models warm

By default Ollama unloads a model 5 minutes after the last request. The next call then re-reads the whole weights file from disk: fine for a 4 GB model, painful for a 200 GB one. For anything you query intermittently, pin it:

# never unload
curl localhost:11434/api/generate -d '{"model":"qwen3-coder","keep_alive":-1}'
# or globally
export OLLAMA_KEEP_ALIVE=-1

Watch what's loaded and where with ollama ps: it shows the model, its size, and the CPU/GPU split, which is the next thing that bites you.

GPU and VRAM control: the cliff

If a model plus its KV cache doesn't fit in VRAM, Ollama silently offloads some layers to CPU. It still works, just at a third of the speed or worse. That CPU spill is the difference between "usable" and "why is this so slow," and ollama ps showing 100% GPU versus 48% CPU/52% GPU is how you catch it.

Levers:

ollama ps                       # is it 100% GPU? if not, you spilled
OLLAMA_GPU_OVERHEAD=0           # reserve less VRAM headroom if you're close
# pull a smaller quant, or shorten num_ctx, to make it fit on one card

This is the same fit math the calculator runs: weights + KV must clear your card's VRAM, or you need a bigger card / more cards / a tighter quant. If the calculator says a model needs two cards, Ollama on one card will spill. Believe it.

Pick the quant on purpose

Ollama's default tag is usually a q4_K_M (≈4.5-bit). That's the right default, but you can ask for others when quality or VRAM demands it:

ollama pull qwen3-coder:30b-q8_0     # higher quality, ~2x the VRAM of q4
ollama pull qwen3-coder:30b-q4_K_M   # the sane default

q4 is the floor for serious work, q8 when you have VRAM to spare and want the quality back, fp16 for benchmarking only. This maps one-to-one to the calculator's quant switch: flip it there to see what each costs in VRAM and tokens/sec before you download 40 GB.

The API and Modelfiles

The reason Ollama "just works" with editors and agents: it speaks the OpenAI API on localhost:11434/v1. Point any OpenAI-compatible client at it with a dummy key:

export OPENAI_BASE_URL=http://localhost:11434/v1
export OPENAI_API_KEY=ollama

To stop repeating per-request options, bake a configured variant with a Modelfile:

FROM qwen3-coder:30b
PARAMETER num_ctx 32768
PARAMETER temperature 0.2
SYSTEM "You are a terse senior engineer. Code first, prose last."

ollama create coder -f Modelfile   # now `coder` carries those settings

Concurrency and Ollama's ceiling

Ollama will handle a few parallel requests:

export OLLAMA_NUM_PARALLEL=4        # concurrent requests per model
export OLLAMA_MAX_LOADED_MODELS=2   # distinct models resident at once

But understand what it is doing: time-slicing a llama.cpp server. It is built for you, or a small team, not a product. The moment you're serving real concurrent users, the calculator's team (20) or startup (200) presets, you want continuous batching, which Ollama doesn't do and vLLM and SGLang do. They read the weights once and apply them across a whole batch of in-flight requests, which is why their aggregate tokens/sec is multiples of Ollama's on the same GPU.

The rule of thumb: Ollama for development and one-to-a-few users; vLLM (or SGLang/TGI) when you're serving a team and throughput per dollar starts to matter. The serving stack changes your tokens/sec several-fold on identical hardware: the serving deep-dive is the whole story.

The loop

Size the box on the cost calculator → pick the model for that memory from the tier picks → get it running with the quickstart → tune it with the knobs above → graduate to vLLM when you're serving more than yourself. Local stopped being a compromise a while ago; it's just a few settings most people never touch.