Vision models for UI and screenshot understanding: an evaluation set for visual reasoning
A vision-language model never actually sees your screenshot. It sees whatever survived the resize, crop, and tiling the runtime applied before a single pixel became a token, and every answer downstream gets built on that transformed version, not the one on your screen.
That gap is why UI evaluation needs its own test set, not a borrowed general-image one: app screens, dialogs, regressions, where a control's position and a line of tiny text carry the answer. Keep the task narrow: identify a control, explain one failure, propose a verifiable next interaction, given the screenshot, viewport metadata, task state, and accessibility data where you have it. Check the DOM or accessibility tree first; pixels earn their place once layout, ambiguity, or a relationship defeats that baseline. The set itself needs glare, blur, occlusion, bad crops, tiny text, not clean demo captures.
What survives the resize
| Field | Why it's logged |
|---|---|
| Model, runtime, prompt revision | ties result to one setup |
| Crop, resize, orientation | shows what reached the model |
| Image count, order, timestamps | catches identity, sequence errors |
| Token accounting, latency, memory | separates model cost from pipeline |
| Grounded pass, unsupported claim, abstain | scores confidence, not accuracy |
If the provider exposes an image-detail setting, log the value actually used. Local runtimes need one more line: which projector paired with which checkpoint, at what quantization. A mismatch there stays quiet, the server keeps answering fluently while the grounding underneath has broken.
Two ways a screenshot answer goes wrong
The model invents a control that isn't visible, or clicks the right label in the wrong region. Both slide past a quick review because they match what a reviewer expects, and tidy benchmark shots hide the gap too. Demand a region, frame, or label per conclusion, shown beside the answer.
Split the reasoning chain: "a red indicator, top right" is observation, "it belongs to the network panel" is association, "the connection failed" is inference, "restart the gateway" is an action needing more evidence than the other three combined. Don't let a plausible observation walk into an irreversible tool call.
Multiple images add identity risk: label each one, say whether they're the same object or a sequence, drop duplicates, shuffle frames when order matters. Same story either way means it isn't using the sequence.
Score unsupported claims apart from omissions: a missing detail gets flagged, an invented control or identity can trigger an automated action nobody checked. Include real unanswerable examples and reward the model for saying so.
The line between hosted and on-disk
Privacy runs both directions: crop faces, screens, addresses where you can, and set retention separately for originals, thumbnails, OCR text, and debug captures. Running local cuts external transfer. It does nothing for an unencrypted disk or an overbroad dashboard account.
Track a hosted path like gemini-cli-multimodal on upload time and rate limits; track a local one like ollama-vision-local on cold load and concurrency. Compare the whole path at p50 and p95, not tokens per second alone, and stop tuning once the gate clears at acceptable latency and abstention. A bigger model mostly leaks more private detail and answers past the evidence.
Next thing worth checking on any pipeline you didn't build: does the runtime log which projector shipped with which checkpoint. If not, you're flying blind on the mismatch that keeps producing fluent, wrong answers.