Vision models for UI and screenshot understanding: video through frame sampling

Here's the rule I open every screenshot-and-video vision task with: sample frames at event boundaries, not on a fixed clock, and only bring in a video-native model when the sequence between frames is the thing you're testing. The rest of this is me making the case for why I trust it enough to ship.

the screenshot isn't what the model sees

The image feels like solid ground because you can see it, but the model never receives your screenshot. It gets a transformed version: a resolution, a crop, a tiling scheme, a token budget chosen for cost. A fluent answer can be fully grounded, partially grounded, or built around a detail cropped away before inference ran, and the answer's tone won't tell you which.

For UI and screenshot work, the material is app screenshots, browser states, dialogs, and visual regressions. The job is narrow: name the visible controls and state, explain a failure, or propose a next interaction you can check. Feed the model a screenshot plus viewport metadata, task state, accessibility data if you have it, and the expected outcome, not a bare image and a question. A narrow question is one you can score.

Try the boring path first: DOM or accessibility-tree inspection, pixels only for what those interfaces omit. Vision-language reasoning earns its keep on layout, ambiguity, relationships, or language fixed rules can't handle economically.

get the pixel pipeline sorted first

Sort out what happens to the pixels before you write a prompt line. Pick frames on motion or scene-change signals instead of a stopwatch, keep the frame before and after it, and check the selection against clips where you already know what happened. Keep the original asset untouched and log every transform you ran before the model saw the pixels. A model name in your eval log tells you almost nothing on its own.

model: id + api/runtime + prompt revision
image: original dims, crop, resize, orientation
frames: count, order, timestamps
context: OCR/metadata supplied beside pixels
tokens: visual/input/output accounting
timing: first response, total latency, peak memory
verdict: grounded / unsupported claim / abstain / repair

If the provider exposes an image-detail or resolution control, log the value you got, not the default. Running this locally, say through something like Ollama's vision models, means logging the vision projector, quantization, context setting, and the preprocessing path. Pair a text checkpoint from one release with a projector from another and the server keeps answering fluently while quietly getting the picture wrong, and nothing in the response tells you that.

the label can be right and the click still wrong

The acceptance gate is grounded element identification, coordinate accuracy, state interpretation, and whether the action survives a browser check. Score a missing answer separately from a made-up one. Missing information just means look harder. An invented serial number, control, trend, identity, or product attribute drives a wrong action while looking as confident as a correct one. Put genuinely unanswerable cases in the eval set and reward the model for saying so.

The failure that survives review is the model inventing a control that was never on screen, or naming the right label while pointing at the wrong part of the layout. The sampling-specific version is picking frames on a fixed, slow interval and then blaming the model for missing something between two of them. Both slip past a quick read because the answer matches what a reviewer expected. Require the region, page, frame, or label behind each conclusion, and show it to the reviewer instead of a log.

Split the prompt into four layers, each needing different evidence: observation, association, inference, action. A red indicator upper right is observation. That it belongs to the network panel is association. That the connection has failed is inference. Restarting the gateway is action, and it shouldn't follow straight from an observation.

Multiple images bring identity risk. Say whether two frames show the same object, the same page in sequence, the same camera, or the same time; the model won't infer that on its own. Cut duplicate frames, they spend tokens without adding evidence. Where order matters, attach timestamps and run a shuffle test: a model telling the same causal story after you scramble the frame order isn't using the sequence.

two bills that arrive after the answer

Privacy is a before-and-after problem, not a during-inference one. Crop out the rooms, faces, other screens, addresses, and reflections that don't belong in the eval set before it reaches the model. Set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures; each carries a different risk if it leaks. A local model cuts down what leaves the building. It does nothing for an unencrypted disk or a dashboard account with more reach than it needs.

The second bill is latency, and it's easy to under-measure. For a hosted route, say through something like Gemini CLI's multimodal handling, fold upload time, regional routing, retention settings, rate limits, and provider errors in alongside answer quality. Running locally, watch cold load time, projector memory, CPU-versus-GPU placement, preprocessing time, heat, and concurrent load. Measure the whole path at the 95th percentile, not the median, and not just tokens per second, because the tail is where a bad sampling decision shows up.

Stop tuning once the gate passes at a latency, cost, and abstention rate you can live with. A bigger model and more pixels are not automatically safer: they can expose more of what should have stayed cropped, and make it easier to answer past what the evidence supports. So the rule I opened with stands. Sample around the event, not the clock. That's the whole rule.