Budgeting VRAM for local vision models

Point a 14B-class vision-language model at a folder of screenshots and PDF pages, and the memory picture stops resembling the text-only chatbot next to it on the same GPU.

Before the model writes a word

Every image gets sliced into a grid, run through a vision encoder, and turned into projected tokens that land in the context window before generation starts. One screenshot can outweigh a page of dense prose, and history that accumulates across turns grows past what a text-only VRAM rule of thumb expects.

Start from one box, one named model, one fixed prompt set, and the exact launch command written down. Vary resolution and image count per turn, watch the visual token count, then cap how many images a follow-up turn keeps, since unbounded history is how a fine demo falls over by lunch. Ollama's vision path hits the same trap regardless of runtime.

Measure the request across the whole system, not as one number.

What that number means depends on the job: runs per hour for a batch, or the waits a person notices for anything interactive.

The failure that never throws an error (vision)

The mistake that survives longest is sizing from the text-model quantization alone, then meeting the real image load once it's live. It survives because nothing crashes.

The model keeps answering while the failure shows up as behavior: partial CPU offload (see the GPU offload math), an unlogged cache miss, swap creeping up, a fallback shipping the image to a backend you didn't choose.

Watch the runtime log and memory counters while a test runs, not just the final number. Change one variable at a time, repeat enough to rule out a lucky sample, and read the output instead of trusting speed alone.

Boring matters too: hand-repair after every driver update is not free, and a reproducible config beats a clever one you can't rebuild. Skip the raw artifacts, keep the hashes and commands, so a rerun in six weeks means the same thing.

Size for the worst turn you'll actually see

Budget from the largest real multimodal turn you expect, not a thumbnail from a demo. Less flattering than the newest accelerator, but it leaves limits you can see coming.

A visible limit gets scheduled or routed around. An invisible one turns into a middle-of-the-night page and an order nobody planned.

Stop adding headroom once quality and latency hit target with margin left over. That margin absorbs the next longer document, the next user, the next runtime release, not capacity wasted.

The rule I'd keep, if I kept exactly one: budget for the ugliest turn a user will actually send, not the one in the screenshot you took for the README.