Vision models for document vision: privacy and security for visual inputs
Local inference does not make a vision pipeline private. It only relocates the leak: the image sits on a disk, in a log, in a backup with no expiry date.
The picture looks whole, but the model works from a resized, cropped, tiled version with its own token budget, and a fluent answer can be grounded, partly grounded, or built on a detail trimmed away before inference started.
Narrow the job to document work and this stops being abstract: scanned forms, invoices, reports, mixed-layout PDFs, pulling fields, tables, layout relationships, and source locations without losing qualifiers. Feed the model the page image, existing OCR text, the page number, and the target schema. Try OCR plus deterministic parsing first; bring the model in once that baseline chokes on layout, ambiguity, or language fixed rules can't cover economically.
What survived the resize is what you're grading
Map every stored frame and thumbnail before you write a prompt: crop or redact before upload, isolate tenants, treat visible image text as untrusted. Keep the original asset, log every transformation, then log the run: model ID, runtime, prompt revision, original dimensions plus crop, resize, and orientation, image count and order, what OCR or metadata rode beside the pixels, token accounting by visual, input, and output, first-response and total latency, peak memory, the image-detail value sent, and a verdict of grounded pass, unsupported claim, abstention, or repair. For local models, log the vision projector, quantization, context length, and preprocessing code; a mismatched projector fails quietly; the server answers fluently.
Grade on field accuracy, table structure, source coordinates, abstention rate, and review time per page. Score an unsupported claim separately from a missing one: an omission gets caught at review, but an invented serial number, control, trend, identity, or attribute can slide into a decision nobody rechecks. Put unanswerable pages in the set and reward abstaining.
Don't let a caption authorize the next step
The model always answers. Whether it saw what it describes is a separate question.
The failure that survives review looks right: a plausible value copied from the wrong row, page, or section, right where a reviewer expects the real one. Split every claim into observation, association, inference, and action; each needs different evidence and trust. Require a region, page, or feature behind each conclusion, and show it to the reviewer, not a log. A prompt is advice. Code is a wall. Never let image text authorize a tool call on its own.
Multiple images add an identity problem: label each one, note whether they're the same object, a page sequence, or the same camera, and drop duplicate frames. Attach timestamps if order matters; the same story after shuffling means the model isn't using the sequence. Retention should differ for originals, thumbnails, OCR text, embeddings, and debug captures; crop irrelevant faces, screens, and reflections before anything leaves the device. Whether hosted through Gemini CLI's multimodal mode or local through Ollama's vision support, measure the path at p50 and p95, not tokens per second. Stop once the gate passes at acceptable cost and abstention; bigger isn't automatically safer, just more surface to leak through.
I still don't have a good automated check for whether crop-and-redact actually ran before an image left the device, just a code-review habit, not a test I can point to. And I wouldn't bother with the four-layer split on a simple invoice; save it for cases where a guess could quietly become an action nobody approved.