Vision models for UI and screenshot understanding: reasoning across multiple images
People have already shipped this exact mistake to production. Feed a vision-language model a stack of screenshots from a broken build and ask why a dialog looks wrong, and it answers fast and specific, even when the detail belongs to a different screenshot entirely. Nobody catches it on the first pass because it reads like something a person looking at the same images would say. A bug report gets filed against the wrong element, a fix lands that changes nothing, and the regression survives another release cycle.
The reason this keeps happening is that a screenshot feels concrete in a way that misleads people about what the model actually received. You see pixels. The model sees a resized, cropped, tiled representation squeezed into a token budget, and a fluent answer can be grounded in that, half-grounded, or built around a detail dropped before inference even started. For UI and screenshot work, the material is app screenshots, browser states, dialogs, visual diffs, and the job is narrow: name the visible controls and state, explain a failure, or propose one verifiable next step. Give it the screenshot plus viewport metadata, task state, accessibility data if you have it, and the expected outcome. That's a narrower ask than "understand this image," and it's the version you can actually score.
Before any of that, check whether you need a vision model at all. DOM and accessibility-tree inspection answers most UI questions without touching pixels, cheaper and more reliable when it applies. Reach for a VLM once the baseline hits layout, ambiguity, cross-element relationships, or free-text language fixed rules can't handle without turning into busywork.
What you log before you trust a single answer
Once the baseline can't do the job, the setup work is what makes the output worth reading. Preserve the original image, record every transformation applied before the call, label each input, keep timestamps, and note page or camera identity so conclusions survive reordering. A useful evaluation record tracks:
- model ID, API or runtime, and prompt revision
- original dimensions, crop, resize, orientation
- image count, ordering, and frame timestamps
- token accounting across visual, input, and output
- outcome bucket: grounded pass, unsupported claim, abstain, or repair
If the provider exposes an image-detail or resolution setting, log the value actually used, not the default. Running something local, say through Ollama's vision support, record the vision projector, quantization, and preprocessing too: text weights from one build paired with a projector from another can fail quietly while the server keeps returning fluent language. Score unsupported claims separately from missing information. An omission gets flagged for review; an invented control, serial number, or product attribute can drive a wrong action with nobody noticing. Include a few genuinely unanswerable examples and reward the model for saying so.
Where two images become one made-up scene
The failure to watch for is a model inventing a control that isn't on screen, or clicking the right label in the wrong region. Hand it several images at once and a second failure shows up: it merges different pages or moments into one confident description that still sounds like something a person would say. Require a region, page, or visible feature behind every conclusion that matters, and show that provenance to whoever reviews the answer instead of burying it in a log.
Split the reasoning into layers: "red indicator, top right" is observation, "on the network panel" is association, "the connection failed" is inference, "restart the gateway" is action, each needing its own evidence before a plausible observation jumps straight to something you can't undo. With multiple images, state whether they show the same object, page sequence, or time period, and drop duplicate frames that burn tokens without adding evidence. If order matters, run the set shuffled: a model with the same causal story regardless of sequence isn't using it.
Turn that rule into code, not prompt wording: reduce the image set, label each view, permit comparisons only across images with shared identity. Crop faces, screens, and addresses out of anything you don't need, and set retention separately for originals, OCR, and debug captures; a local model cuts external transfer but does nothing for an unencrypted disk or an overbroad dashboard account. Whichever way you drive it, hosted API or Gemini CLI's multimodal handling, measure the whole path, upload time included, at p50 and p95, not tokens per second alone, and stop tuning once the gate passes at acceptable cost. A bigger model with more pixels isn't automatically safer, it can just as easily expose more private detail and answer past the evidence.
None of this stops a model from being wrong with total confidence anyway. It just makes the wrong answers easier to catch before someone acts on one.