Vision models for UI and screenshot understanding: preprocessing before the vision model

And that's the trap: the screenshot on your monitor and the pixels the model reasoned over are two different objects, related by a resizer, a tiler, and an unlogged token budget. The answer comes back fluent, so you can't tell whether it's grounded, half grounded, or built on a detail cropped out before inference. For UI and screenshot work that gap costs more: name what's visible, explain a failure, propose the next click. Narrow the job first. A screenshot plus viewport metadata, task state, accessibility data, and expected outcome beats "understand this image," the only version you can measure.

Preprocessing should strip irrelevant variation, not evidence, not meaning. Ask first whether you even need a vision model. DOM and accessibility-tree inspection tell you what's on a page; pixels earn their place once layout or ambiguity defeats that route.

build the eval before you touch the model

Version a deterministic preprocessing pipeline: compare raw, normalized, cropped, and OCR-augmented inputs against each other, keeping the original and a record of every transform. That record needs the model ID, prompt revision, dimensions and crop, image count and order, token accounting, and whether the answer landed grounded, unsupported, abstained, or repaired. If the provider exposes an image-detail setting, log the value it used, not the default you assumed. Do the same for a local model through Ollama: log the projector and quantization, since a mismatch degrades quietly while it keeps talking. The gate is grounded element identification, coordinate accuracy, state reading, and a next step that verifies in the browser, with "I don't know" scored apart from "I made something up": a missing detail earns review, an invented control walks into a wrong action; seed the set with unanswerable examples and reward saying so.

what breaks once it's running for real

The characteristic failure: inventing a control that isn't there, or clicking the right label in the wrong region. The screenshot-specific trap is aggressive sharpening that looks clean but invents a character or hides a defect, and both slip past review since they match what's expected, so require a region or label behind every conclusion and show it to the reviewer. Separate the layers: observation, association, inference, action; a plausible observation shouldn't jump straight to the last one. Multiple images add identity risk: label them, say if they're the same object or moment, drop duplicates, and when order matters, test a shuffled run, since an unchanged story means the sequence isn't in use. Keep originals retrievable, transforms reproducible in code not prompt text, and crop rooms, faces, and addresses before and after inference; vary retention for originals, thumbnails, and captures, since local doesn't secure an unencrypted disk. Weigh hosted paths, Gemini CLI's multimodal mode included, against local ones at p50 and p95, not tokens per second, then stop once the gate passes on latency, cost, and abstention. A bigger model and more pixels aren't automatically safer; they can expose more and invite answers past the evidence.

Go pull the image-detail value your last few calls actually used and check it against what you assumed the provider was sending. That's usually where the story falls apart.