Vision models for document vision: local, hosted, and hybrid vision deployment

Somebody has already rubber-stamped a wrong total because a vision model read the number off the row above and stated it like a fact. Somebody else has already had a scan with a customer's home address in the corner shipped to a third-party API that never needed to see it. Neither was careless. An ungrounded answer looked exactly like a grounded one, and nothing told them to check twice.

The input feels concrete because you can see the image, but the model works from a resized, cropped, tiled version with its own token budget, and whatever corner it dropped stays invisible to you. A fluent answer says nothing about whether the detail behind it survived the trip.

For document work the material is scanned forms, invoices, reports, and PDFs with tables stacked inside other tables. The job stays narrow on purpose: extract text, fields, tables, relationships between them, and where each answer came from, without losing the qualifiers that change meaning. Feed the model page images next to OCR text, page numbers, and the schema you want filled in. A bounded question like that is the only kind you can route, evaluate, and fail safely on.

So route on sensitivity, image size, how hard the layout is, where the hardware sits, and the deadline. Start cheap: OCR plus deterministic parsing clears most well-formed pages alone. Bring a vision-language model in once the baseline hits a layout, an ambiguity, or a cross-field relationship fixed rules cannot untangle at a price worth paying.

Log the pixels before you trust the sentence

Run the same visual gate locally, through something like Ollama's vision support, and against a hosted model driven through the Gemini CLI's multimodal mode. Put upload time and review inside the clock, decide the escalation condition ahead of time, and keep the original image untouched with every transformation logged. A model name in a spreadsheet tells you almost nothing; the record needs:

• model, runtime, prompt revision
• dimensions, crop, resize, orientation
• image count, order, frame timestamps
• token counts and total latency
• grounded, unsupported claim, abstain, or repair

If the provider exposes an image-detail setting, log the value used, not the default. Running local, log the vision projector, quantization, context length, and preprocessing code: weights from one release paired with a projector from another can misbehave without throwing an error. The server just keeps talking. It talks about the wrong page.

Score against field accuracy, table structure, source coordinates, correct abstention, and review time; keep unsupported claims separate from missing information. A blank field sends someone back to check; an invented serial number or identity can send someone into a wrong action. Put unanswerable pages in the test set, and score the model up for admitting it does not know.

The row it actually came from

The failure that shows up most is a plausible field value pulled from the wrong row, page, or section, formatted so cleanly you'd never guess. It hurts two ways: a local model that quietly escalates a sensitive image the moment it stalls, or the opposite, every thumbnail routed to an expensive frontier model regardless of need. Both pass a casual look, because the output matches what the reviewer already expected.

Make the model name a region, page, frame, or visible feature for every conclusion that matters, and show that provenance to the reviewer instead of a log nobody opens.

A pipeline collapses everything into one confident sentence by default, so separate what the model is claiming into layers. "There's a total in the bottom right" is observation. "It belongs to the shipping line" is association. "The invoice is short one unit" is inference. "Issue a credit" is an action. Each layer needs different evidence and trust, and an observation should never skip straight to an irreversible action.

Four separate claims wearing one sentence

Hand the model more than one image and you've added an identity problem on top of the grounding one. Label each input and say whether they show the same object, page sequence, camera, or time period, because the model will happily invent a relationship you never stated. Strip duplicate frames before they burn tokens without adding evidence. Where order matters, attach timestamps and run a reordering test: if shuffling the frames does not change the causal story, the model was pattern-matching, not using the order.

Put the rule in code, not in the prompt

Keep privacy rules hard, keep escalation visible, and compare accepted visual tasks per dollar and per second, not by model reputation. Write that rule into preprocessing, routing, and authorization code: prompt instructions are a request, not a control, and requests are the first thing an under-pressure pipeline forgets. Store the model and pipeline version with every result so an upgrade can be replayed against the same images.

Crop rooms, faces, screens, addresses, and reflections that do not belong in the answer, before and after inference. Set retention separately for originals, thumbnails, OCR text, embeddings, and prompts, since each leaks differently. A local model keeps pixels off someone else's server, worth something, but it does nothing for an unencrypted disk or an overbroad dashboard account.

Test the hosted path on upload time, regional routing, retention, and rate limits; test the local path on cold load, projector memory, GPU placement, and concurrency under heat. Compare the whole path at p50 and p95, not the tokens-per-second figure vendors lead with. Stop tuning once the gate passes at a latency, cost, and abstention rate you'd sign off on. A bigger model and more pixels are not automatically safer; they expose more private detail and encourage answers past the evidence.

I run anything with a name, an account number, or a face on it through a local model, full stop, and I've made peace with what that costs me. On the gnarly layouts a hosted model would probably nail the table structure more often than mine does, and I'm giving that up on purpose. I'd rather review more failures than explain one leak.