Vision models for chart and diagram understanding: privacy and security for visual inputs
A model reading your chart doesn't lie, it answers honestly about a different picture: a resized, cropped, tiled rendering, sliced to whatever token budget the runtime allowed. Every sentence back describes that rendering, not your file.
Narrow the job first. The territory is dashboards, plots, architecture diagrams, schematics, slide visuals: recovering a visual relationship and answering one precise question. Feed it the original image or a vector export, the nearby caption, legend, units, and the exact question. Check whether the chart's underlying data already exists; pixels are the fallback, not the opening move.
the pipeline decides what the model sees
Visual privacy is plumbing: collect less, redact early, set retention on purpose, control access, and treat text inside an image as untrusted input, never an instruction. Map every stored frame and thumbnail, crop or redact before upload, isolate tenants, and log every transformation before the model call.
Keep an evaluation record broader than a model name: runtime and prompt revision, crop history, frame timestamps, token accounting, latency, and a verdict: grounded, unsupported, abstain, or repair. Log the resolution setting a provider actually used, not the default, whether you're driving Gemini CLI's multimodal mode or a local model through Ollama's vision path, where you also own the projector and the quantization. Mix text weights from one release with a projector from another and the pairing fails quietly: the server keeps talking, it just stops describing your image.
a convincing answer and a true one are different claims
Score label transcription, relationship accuracy, numerical consistency, evidence citation, and abstention on the unreadable, and score unsupported claims apart from missing information. An omission gets caught in review. An invented serial number or a control that isn't there walks straight into an action nobody double-checks.
The failure that survives review is a confident story: axis reversed, wrong series, wrong direction on an arrow, sounding right because it matches what a person expected.
Local doesn't mean forgotten. It means the leak moved to a different disk.
That's the other trap: calling a system private because inference runs locally while raw images sit indefinitely in logs and backups nobody rotates. Require a region, page, or visible feature behind every conclusion, shown to the reviewer, not just the log. Separate observation from association from inference from action, and don't let a plausible observation walk into an irreversible tool call. With more than one image, label each explicitly, say what they show, and drop duplicates; if order matters, test a shuffled version, because a model telling the same story either way isn't using the sequence.
keeping it honest after the demo ends
Capture the smallest useful region, keep it briefly, and never let text inside an image authorize a tool call. Put that in preprocessing and authorization code, not prompt instructions someone can quietly delete, and store pipeline versions so an upgrade can be replayed.
Crop what you can before the model sees it: rooms, faces, screens, addresses, reflections. Set retention separately for originals, thumbnails, OCR text, embeddings, and prompts. A local model cuts external transfer, not an unencrypted disk or an overbroad dashboard account.
Test hosted and local paths separately: upload time, regional routing, and retention for one; cold load, projector memory, and device placement for the other, compared at typical and worst-case latency, not tokens per second alone. Stop optimizing once the gate passes at a cost you can live with. A bigger model and more pixels aren't automatically safer; they can expose more of the frame.
What I still don't have is a way to verify a zero-retention setting actually behaves like zero, on either side of that hosted-versus-local line, instead of a checkbox nobody enforces.