Vision models for UI and screenshot understanding: resolution and visual-token budgets
The image you upload and the image the model reasons over are not the same object.
By the time a screenshot reaches inference it has been resized to whatever the vision encoder trained on, tiled if large, and turned into a fixed count of visual tokens competing with your prompt for the same context window. None of that shows in the response, and past a point of confidence you cannot check what the model actually looked at.
The material is application screenshots, browser states, dialogs, visual regressions. The job: name the visible controls, explain a failure, propose a next click you can verify. Feed the pipeline more than pixels: viewport size, task state, the accessibility tree when you can get one, and what correct looks like. That is the task, not "understand this image."
Resize first, prompt second
Decide what the model is allowed to see before writing a prompt, and whether it needs pixels at all. Plenty of UI state already lives in the DOM and the accessibility tree; pull that first, and reach for vision only where those fall short. Useful resolution is set by the smallest evidence the task has to resolve, not by how many megapixels the screenshot tool captured.
Keep the original file untouched and log every transformation before the call: crop, resize, orientation, tiling, the actual resolution setting the API used. A local model raises the stakes: the vision projector, quantization, and context length are separate from the text weights, and a mismatched projector degrades quietly while the server keeps returning fluent answers.
A row per screenshot, not per run
Run the same sweep you would run on any model change, dimensions and crop strategy against accuracy, and log more than which model you called.
| Capture this | Why it matters |
|---|---|
| Model, API/runtime, prompt version | Reproducible config |
| Size, crop, image count/order | What the model actually received |
| Tokens: visual, input, output, cost | Preprocessing spend vs. generation spend |
| First-token and total latency, peak memory | What a user actually feels |
| Grounded, unsupported, abstained, repaired | Different failure, different fix |
The bar is grounded element identification, coordinate accuracy, correct state interpretation, and an action that verifies. Grade unsupported claims apart from missing ones: saying it cannot tell is annoying, inventing a control or value nobody put there drives a wrong action nobody notices. Put a few unanswerable screenshots in on purpose; abstaining should score as a pass.
Two ways to lose the text
Two failure shapes look nothing alike from outside. One is downscaling a whole page until label text turns to mush and the model guesses at a control, or invents one never rendered. The other is uploading a full, high-resolution capture for one small corner, burning tokens on background nobody asked about while still landing the click a few pixels off. Both read fine, because they land close to expectation.
Resolution alone does not fix this; a named region, frame, or visible feature behind every conclusion does, and that provenance needs to reach the reviewer, not a log nobody opens.
Don't let a guess click a button
Separate what the model is doing at each step: noticing something, tying it to a component, drawing a conclusion, then acting. "Red badge in the corner" is an observation. "That's the sync indicator" is an association. "Sync is failing" is an inference. "Restart the sync service" is an action needing a higher bar of evidence. Never let it jump straight from a pixel to a tool call.
More than one screenshot adds a second problem: which image is which, and whether order matters. Label them, say plainly whether they show the same page, session, or a sequence over time, and cut duplicate frames before they eat tokens for nothing. If order matters, attach timestamps and shuffle-test; a model telling the same story regardless of order is pattern-matching, not reading it.
Hosted and local cost different things
Put the crop-to-evidence rule into preprocessing or validation code, not prompt wording someone edits later, and store the model and pipeline version with every result so an upgrade replays against the same images.
The two paths fail differently too. Hosted means upload time, regional routing, retention policy, and rate limits; Gemini CLI is a fast way to sanity-check that path before production. Local means cold-load time, projector memory, GPU placement, and how hot the box runs under concurrency; Ollama's vision models make it cheap to swap projector and quantization pairs without rebuilding anything. Measure both at p50 and p95, not just tokens per second.
Privacy sits on both sides of the request: crop out rooms, faces, and addresses that never needed to travel, and set separate retention for the original, the thumbnail, OCR text, embeddings, and debug captures. A local model stops the upload; it does nothing for an unencrypted disk or an overbroad dashboard account.
Stop pushing resolution and model size once the task clears its accuracy bar at a latency and cost you can live with. Bigger images and models aren't a free safety upgrade; they're more surface for private detail and more room to answer past what the picture shows.
Crop to the evidence, keep just enough pixels to support it, cap everything else. That's the rule.