Vision models for UI and screenshot understanding: choosing the right vision model

Here's the rule: don't point a vision model at a screenshot until the DOM and the accessibility tree have failed you. Most of what looks like a vision problem in UI testing is a solved problem in structured data.

Once pixels are necessary, pick the model on text density, spatial reasoning, image count, language, latency, and deployment constraints, not leaderboard rank. Deploy the smallest one that clears your hardest case and abstains on the rest, wired into code, not prompt prose.

An image feels like solid ground, everyone can see it, but the model sees a resized, cropped, tiled version squeezed into a token budget. A fluent answer can be fully grounded, half grounded, or built on a detail cropped out before inference started.

The material here is app screenshots, browser states, dialogs, and visual regressions: name visible controls and state, explain a failure, or propose one verifiable next click, fed the screenshot, viewport metadata, task state, accessibility data, and the expected outcome.

keep the receipts from pixel to verdict

Build the evaluation before touching a prompt: examples split easy, ambiguous, impossible, run at the exact API call or local runtime you'll ship. Keep the original file untouched, log every transformation before the model sees it. The record needs more than a model name:

eval_row:
  model: id, api/runtime, prompt_rev
  image: dims, crop, resize, orientation
  batch: count, order, frame_ts, ocr_alongside_pixels
  cost: tokens(visual/in/out), latency, peak_mem
  verdict: grounded | unsupported | abstain | repair

If the provider exposes an image-detail or resolution flag (gemini-cli-multimodal covers the mechanics), log the value actually used, not the default. Running local? Read ollama-vision-local: the quiet failure is a mismatched text stack and projector, or wrong quantization, while the server keeps answering fluently regardless.

Grade against four things: grounded element identification, coordinate accuracy, state interpretation, and whether the browser action verifies. Score an invented detail apart from a missing one: a gap sends a case to review, an invented control or trend sails past and drives the wrong click. Seed unanswerable examples, and reward abstaining.

two ways a plausible answer lies to you

The failure has two shapes: an invented control, or the right label in the wrong region, both surviving casual review because the answer looks convincing. The adjacent trap is a flagship model when OCR, embeddings, or a small task model clears the gate cheaper. Require a region, frame, or feature behind every conclusion, shown to the reviewer, not buried in a log.

Split the prompt into layers: observation ("red indicator, top right"), association ("belongs to the network panel"), inference ("the connection failed"), action ("restart the gateway"). Each needs its own evidence, and a plausible observation shouldn't jump straight to an irreversible tool call.

Multiple images add identity risk: label each one, state whether they're the same object, a sequence, a different camera, or a different time, and cut duplicate frames that add nothing. If order matters, attach timestamps and shuffle the frames: a model telling the same story either way wasn't using the sequence, it was pattern-matching a script.

run the full path, not the demo

Wire the smallest-path rule into preprocessing, routing, and validation code; store model and pipeline version with every result so upgrades replay later. Privacy applies both directions: crop faces, screens, addresses you don't need; retain originals, thumbnails, and OCR text separately, lifespans differ. Local cuts data leaving the building; it doesn't fix an unencrypted disk or an overbroad dashboard account.

Test hosted models on upload time, regional routing, retention, rate limits, and failures. Test local ones on cold-load time, projector memory, GPU placement, preprocessing time, heat, and concurrency. Compare the whole path at p50 and p95: tokens-per-second alone says little about wait time.

Stop tuning once the gate passes at a latency, cost, and abstention rate you can live with. A bigger model and more pixels aren't automatically safer: mostly more private detail in the pipeline, and more temptation to answer past the evidence.

I'll admit the harness is more machinery than most screenshot tasks need. Eyeballing one dialog a week? Skip it, look at the picture. The day that stops working is the day you needed this, and you won't notice until it's cost you something.