Vision models for UI and screenshot understanding: structured output from images
A vision model will tell you the login button is red when it's grey, and wrap that wrong answer in valid JSON. Getting schema-shaped output is easy now, almost boring. The schema says nothing about whether a value came from the pixels you sent or from a cropped, resized, tiled version that got mangled before inference started.
For UI and screenshot work the task is narrower than "understand this image": application screenshots, browser states, dialogs, visual regressions, where the job is naming visible controls, explaining a failure, or proposing one verifiable next click. Feed the model a screenshot with viewport metadata, task state, accessibility data where available, and the expected outcome, which is what makes model choice and evaluation measurable, not eyeballed.
Start from the DOM, not the pixels
My decision rule: pixels are the fallback, not the default. Use accessibility-tree or DOM inspection when it answers the question, and reach for a vision-language model only when layout, ambiguity, relationships, or on-screen language make the fixed-rule route too expensive. Not caution for show: a schema constrains the shape of an answer, not whether the value was read off the image or guessed from what a UI usually looks like.
A model that tells the same story after you shuffle the frames isn't reading the sequence, it's pattern-matching the vibe of one.
What the eval record actually needs to hold
An evaluation record needs more than a model name on a verdict: model ID, runtime, and prompt revision; original dimensions plus crop and resize; frame count and timestamps for multi-frame shots; OCR text beside the pixels; token counts across visual, input, and output; latency and peak memory; and the outcome, graded grounded pass, unsupported claim, abstain, or repair, with a pipeline version pinned for replay. Log the image-detail setting used, not the default, on Gemini's CLI multimodal mode or a local model through Ollama alike, plus projector, quantization, and context length locally, since a mismatched build and projector can fail quietly while the server returns fluent nonsense. The bar for passing: grounded element identification, coordinate accuracy, correct state, and a live browser check. Score unsupported claims separately: an omission gets flagged for review, an invented control or trend walks into a database. Build unanswerable cases into the set, and reward admitting that over guessing.
Why the confident answer is the dangerous one
The failure that matters isn't a garbled response, it's a fluent one built on a control that was never there, or a click on the right label, wrong region. Require a region, frame, or feature behind every real conclusion, and show it to the reviewer, not a log. Split the reasoning into layers: "there's a red indicator top right" is observation, "it belongs to the network panel" is association, "the connection failed" is inference, "restart the gateway" is an action, each needing different evidence. Multiple images add identity risk: label what each shows, whether they're the same object over time, and drop duplicates. Privacy applies on both sides: crop faces and addresses before inference, and set separate retention for originals, thumbnails, and debug captures after, since local models fix only the transfer half, not the disk half. Judge the full path at p50 and p95, not tokens per second: upload time and routing for hosted, cold-load and projector memory for local.
I'd rather ship a pipeline that admits "can't tell from this screenshot" more than a demo would, than one that guesses at a button that isn't there. A bigger model and more pixels aren't automatically safer: they mean more private detail exposed and more temptation to answer past what the image shows. That costs coverage: legitimate questions get kicked back as abstentions, and the path runs slower than a bare pixels-in-JSON-out call. I'll take that trade. I've watched enough wrong, confidently grounded answers to know which failure is quieter.