Vision models for document vision: reasoning across multiple images

A vision-language model works inside a fixed token budget, and every image you hand it gets resized, cropped, and tiled to fit that budget before the model forms any opinion about what's on the page.

What comes back can be grounded, half grounded, or confidently built around a detail that never survived the crop, and nothing on your screen tells you which.

Point it at scanned invoices, tax forms, inspection reports, or a stack of mixed-layout PDFs and the job stops being abstract: text, fields, tables, their relationships, and the page location behind each answer, without dropping a qualifier.

Run OCR and deterministic parsing first, before it reaches the model. Bring the model in only where fixed rules choke: layout that won't line up into columns, a relationship between two fields no regex catches, or free text that needs reading. That's the whole decision rule.

What the eval record has to hold

The practical test is boring, which is the point: label every image, preserve timestamps and page or camera identity, and check whether conclusions survive when the same images come back in a different order. Keep the original untouched and log every transformation applied before the model saw the file: a model name alone says nothing about what happened between upload and inference.

If the provider exposes an image-detail or resolution control, log the value it used, not the default you assumed. For a local model, write down the vision projector, quantization, context setting, and the preprocessing code in front of it: text weights from one release paired with a projector from another can fail quietly, the server keeps answering fluently, it just stops answering from the picture.

Make it point at the pixel, every time

The failure that bites is a plausible-looking value copied from the wrong row, page, or section, and it survives review because it matches what a tired reviewer already expected.

Hand the model a pile of frames instead of one document and you get the same trap twice: it merges different objects, pages, or moments into one tidy description, with nothing in the output flagging that a merge happened.

Require a region, page number, frame, or visible feature behind every conclusion that matters, and show that provenance to the reviewer instead of burying it in a log file nobody opens.

Keep observation, association, inference, and action as separate layers in the prompt and the output: what's visible, which field it belongs to, what that means, and what to do about it. Each layer needs a different amount of evidence, and a plausible observation should never jump straight to an action you can't undo.

Multiple images add identity risk on top: label each one and say whether they show the same object, a page sequence, the same camera, or different points in time, and drop duplicate frames that spend tokens without adding evidence. Where order matters, attach timestamps and run a shuffled control; a model that tells the same story after the frames get scrambled was never using the sequence.

The part that doesn't show up in a demo

Privacy work happens on both sides of inference, not just before it. Crop out irrelevant rooms, faces, screens, addresses, and reflections where you can, and set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures: none of them need to live for the same number of days.

A local model keeps the bytes off someone else's network, which is real, but does nothing for an unencrypted disk or a dashboard account with more access than the job needs.

A hosted path through something like Gemini CLI's multimodal mode needs upload time, regional routing, retention settings, rate limits, and provider errors measured, not just token throughput. A local vision model through Ollama needs cold load time, projector memory, CPU or GPU placement, preprocessing time, heat, and concurrency measured, then the whole path compared at p50 and p95, not generation tokens per second alone.

Stop optimizing once the visual gate passes at acceptable latency, cost, and abstention rate. A bigger model and more pixels aren't automatically safer; they can expose more private detail and nudge the system toward answering questions the evidence never supported.

None of this buys you a model that says it doesn't know when it should. Clean logging and visible provenance can still hand back a confident, well-formatted answer to a question the page never answered, and you'll only find that out after someone has already acted on it.