Vision models for UI and screenshot understanding: privacy and security for visual inputs

So no, the model isn't looking at your screenshot. It's reasoning over whatever the resize and crop left behind, and a fluent answer can rest on a vanished detail.

Scope the question down to something you can grade

For UI and screenshot work, the material is narrow: screens, browser states, dialogs, regressions. The job is naming what's visible, explaining a failure, or proposing one verifiable next click. Feed the model a screenshot, viewport metadata, task state, accessibility data, and the expected outcome, not just "understand this image."

The privacy side isn't bolted on, it's the constraint: collect less, redact early, set retention up front, lock down access, and treat image text as untrusted input.

Check what the DOM already knows before you spend pixels

Ask the DOM or accessibility tree first. Most UI questions don't need pixels. Pixels earn their place on layout, ambiguity, or language fixed rules handle badly.

Build the pipeline before the prompt: map stored frames and thumbnails, crop or redact before upload, isolate tenants, and treat instruction-shaped image text as data, not orders. Keep the original and log every transformation.

The evaluation record needs more than a model name: runtime, prompt revision, original dimensions versus crop and resize, image count and order, OCR alongside pixels, token accounting, latency, peak memory, and whether the answer was grounded, unsupported, abstained, or repaired. Log the resolution setting used.

Running locally, log the vision projector, quantization, context window, and preprocessing. A mismatched projector can degrade answers without an error; the server keeps talking about the wrong picture.

Separate what you saw from what you decided to do about it

The bar is grounded element identification, coordinate accuracy, correct state interpretation, and an action that verifies in the browser. Score unsupported claims apart from missing information: an omission gets a second look, an invented control, trend, or identity becomes a wrong action nobody questions. Include unanswerable examples and reward abstention.

The recurring failure is inventing a control that was never on screen, or clicking the right label in the wrong region. The domain-specific trap is calling a setup private because inference runs locally while raw screenshots sit in logs and backups. Both survive review: the answer looks right.

Every conclusion that matters should point at a region, page, frame, or label, shown to the reviewer, not buried in a log.

Four kinds of claim: observation, association, inference, action. A red indicator is observation; that it's on the network panel is association; that the connection failed is inference; restarting the gateway is action. Don't let observation slide into an irreversible tool call.

Multiple screenshots multiply the ways this goes wrong

Hand it more than one image and you've added an identity problem. Label inputs and say whether they show the same object, page sequence, camera, or time period. Drop duplicate frames; they cost tokens, not evidence. If order matters, attach timestamps and shuffle the frames: a model with the same story regardless of order isn't reading the sequence, it's pattern-matching.

Where the local-versus-hosted math actually gets counted

The rule: capture the smallest region that answers the question, keep it briefly, and never let image text authorize a tool call. Put that in preprocessing and authorization code, not prompt prose. Store the model and pipeline version with results so an upgrade can be replayed.

Privacy applies on both sides of the call. Crop rooms, faces, other screens, addresses, and reflections before upload. Set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures. Running locally cuts external transfer; it does nothing for an unencrypted disk or an overbroad account.

For a hosted stack, test upload time, regional routing, retention, rate limits, and provider errors, as with Gemini CLI's multimodal handling. Locally, the friction is cold load time, projector memory, CPU or GPU placement, and heat under concurrency, as with Ollama's vision models. Compare the whole path at p50 and p95, not tokens per second.

Stop tuning once the gate passes at a latency, cost, and abstention rate you can live with. Bigger models and more pixels aren't automatically safer: both leak more private detail and answer past the evidence.

What I haven't solved is the fight between provenance and minimization. Showing the region that justified a claim means keeping that crop, and "keep it briefly" stops meaning much once every claim wants retained evidence. I don't have a retention window that satisfies both.