Vision models for UI and screenshot understanding: local, hosted, and hybrid vision deployment

A screenshot lands in the queue with a one-line bug report: checkout button looks disabled, state says it should be enabled. You hand it to a vision-language model and ask what's going on. It answers fluently and confidently, and that confidence is the first thing to distrust, because you don't know what it saw.

The file you uploaded is not the file it reasoned over. Between your screenshot and the forward pass it gets resized, cropped, maybe tiled, and squeezed into a token budget nobody checked this week. A control at the edge of a large screenshot can shrink to a handful of pixels, and a tile boundary can slice a modal in half without anyone knowing. The model never flags this; it just answers.

Keep the job narrow: identify visible controls and state, explain a failure, or propose a next interaction you can verify against the running app. The input is a screenshot plus viewport metadata, task state, accessibility data when you have it, and the expected outcome, not a picture and an open-ended question. That's what makes model choice and evaluation measurable instead of a feeling.

Check whether you need a vision model at all first. DOM and accessibility-tree inspection is free, deterministic, and already there. Spend pixels only on what those interfaces don't expose: layout relationships, ambiguity between near-identical elements, a screenshot from a system you don't control.

what the model is actually looking at

The deployment decision is a routing decision: sensitivity, image size, task difficulty, where the hardware sits, and how much latency the workflow can absorb. Run the same visual gate locally and against a hosted API, fold in upload time and human review, and write down explicit conditions for when a case escalates. Working that out after the pipeline ships is a bad week.

Keep the original file untouched. Log every transformation applied before the model sees the pixels. An eval record needs more than a model name and a pass/fail column:

model + runtime + prompt version
original dimensions, crop, resize, rotation
image count and order, timestamps if it's a sequence
non-pixel context supplied: OCR text, accessibility tree, metadata
token accounting: image tokens vs text tokens
latency to first token, total latency, peak memory
outcome: grounded pass / unsupported claim / abstain / repaired

Log the resolution or image-detail value you sent, not the default you assumed. Local models hide the same trap differently: the vision projector, quantization, context window, and preprocessing code all need recording separately from the text weights. Pair the wrong projector with the wrong base model and the server keeps returning fluent language while reasoning over something closer to noise. Nothing crashes. That's what makes it dangerous.

Grade the gate on grounded element identification, coordinate accuracy, state interpretation, and whether the fix verifies once you click it. Score unsupported claims separately from plain gaps: a missing detail earns a review step, but an invented serial number, a control that isn't there, a trend line made up, should never get that far quietly. Put unanswerable cases in the test set and reward saying so.

wrong answers that look right

Two failure shapes cover most of what goes wrong. The model invents a control that isn't in the screenshot, or clicks the right label in the wrong region. Both read as plausible on a quick review, because the wording matches what a person expects. Require a cited region, frame, or label for every conclusion that matters, and show it to the human instead of a log.

A second trap cuts both ways: quietly uploading a sensitive screenshot because the local model choked, or routing every trivial thumbnail through an expensive frontier API out of habit. Neither is a technical failure. Both are policy failures in a technical costume, and the fix belongs in routing code, not a line in the prompt.

Split the reasoning into observation, association, inference, and action, and don't let a response skip layers. "Red indicator top right" is observation. "It belongs to the network panel" is association. "The connection has failed" is inference. "Restart the gateway" is action. Each layer needs its own evidence, and a plausible observation should never jump straight to an irreversible action.

Multiple images multiply the identity risk. Label each one, state whether they're the same object, a page sequence, or different points in time, and strip duplicate frames that burn tokens without adding evidence. If order matters, attach timestamps and test what happens when frames get shuffled: a model telling the same causal story regardless of sequence isn't using the sequence.

Hosted and local paths fail differently, so test them on different things:

• hosted: upload time, regional routing, retention settings, rate limits, provider errors
• local: cold-load time, projector memory, CPU/GPU placement, preprocessing time, thermal throttling, concurrency ceiling

Compare the whole path at p50 and p95, not generation tokens per second alone, because that number says nothing about how long the reviewer sat there. Ollama's vision models are the least painful way to get a working projector-plus-weights pair locally; Gemini CLI's multimodal mode buys elastic capacity and a real ceiling on the hard cases, at the cost of every frame leaving the machine.

Privacy is a before-and-after problem, not a during one. Crop out irrelevant rooms, faces, other screens, addresses, and reflections before anything gets sent. Set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures; none of them need the same lifespan. A local model stops the upload. It does nothing for an unencrypted disk or an overbroad dashboard account.

Stop optimizing the moment the gate passes at acceptable latency, cost, and abstention rate. More pixels and a bigger model are not automatically safer: a model that resolves more detail and answers with more confidence is also more likely to describe something it should have left alone, or answer past what the evidence supports.

I won't bother building a universal sensitivity router that scores every screenshot before it leaves. A fixed rule, this queue always stays local, that one can escalate, beats a learned classifier for a decision with this much downside. The tradeoff I take on purpose is local-first for anything touching customer data, even on days the hosted model would clearly have nailed a case the small local one fumbled. That's slower, and occasionally worse, and I'd rather eat that than find out what an uploaded screenshot cost someone.