Vision models for document vision: choosing the right vision model

So the benchmark score doesn't tell you what resolution the model actually saw. It rewrites your image before reasoning about it: resized, cropped, tiled, packed into whatever token budget the provider picked that week. You see a crisp scan; the model works from whatever survived the pipeline. A confident answer can be fully grounded, half grounded, or built around a detail cropped away before inference started.

Narrow the job first. The material is scanned forms, invoices, reports, mixed-layout PDFs; the task is extraction: text, fields, tables, relationships, the page each answer came from, without softening the source's qualifiers. Feed the model page images next to OCR text, page numbers, and the schema you want filled in. That's narrower than understand this image, and only a narrow ask is scorable.

Pick the model on text density, spatial reasoning, images per call, language, latency, and where you can run it. Start dumb: OCR plus deterministic parsing first, a vision model only once that trips over layout or ambiguity fixed rules can't cover.

what you log, not what you swear you did

Build a small blind evaluation before trusting any of this: easy cases, ambiguous ones, a few genuinely impossible, tested at the settings you intend to ship. Log every transformation before the model saw the image: dimensions, crop, image count, ordering, token accounting, latency, a verdict of grounded, unsupported, abstained, or repaired. Drive this through something like Gemini CLI's multimodal mode and log the resolution it actually negotiated, not the default you assume. Local runs add the vision projector and quantization, since weights from one release paired with a projector from another can fail quietly while the server keeps returning fluent language about the wrong pixels.

count the fabrications separately from the blanks

The acceptance gate is field accuracy, table structure, source coordinates, abstention, and reviewer time per page. Score unsupported claims on a different axis than missing information; they aren't the same defect. A gap triggers a second look. An invented serial number or UI control that isn't there can slide into a downstream action unnoticed. Include cases with genuinely no answer, and reward saying so.

the field looks right because it usually is

The failure that actually bites is a plausible value copied from the wrong row, page, or section. Beside it: reaching for a flagship model when OCR or a smaller task-built one clears the same bar for less money. Both survive casual review because the answer matches what a tired reviewer expects. Require a region, page, or label behind every conclusion that matters, and show that provenance instead of burying it in a log. Split the reasoning into layers: red indicator upper right is observation, belongs to the network panel is association, connection has failed is inference, restart the gateway is action, each needing different evidence and authority. A plausible observation should never jump to an irreversible one.

images that don't know they're a sequence

Multiple images multiply identity risk. Label each input: same object, page sequence, or one camera, because the model will assume continuity nobody stated. Drop duplicate frames; they burn tokens without evidence. Order matters. Say so, back it with timestamps, and test a reordered set. A model telling the same story after shuffling isn't using the sequence. It's pattern-matching around it.

the bill arrives after the demo

Deploy the smallest vision path that clears your hardest case and abstains on the rest, and put that rule into preprocessing or routing code, not prompt prose. Version the model and pipeline with every result for replay. Privacy applies both ways: crop irrelevant rooms, faces, and addresses, and set retention separately for originals, OCR text, and debug captures. A local model cuts external transfer but does nothing for an unencrypted disk. Test hosted models on upload time, routing, and rate limits; run it through something like Ollama instead and the bill changes shape: cold load, projector memory, GPU placement, heat, concurrency, before a single token. Compare the whole path at p50 and p95, not tokens per second. Stop once the gate passes at acceptable cost. A bigger model isn't automatically safer; it can expose more private detail and answer past the evidence.

None of this wins the argument in most rooms. Someone reaches for the flagship model because it's the name they recognize, skips the eval, and it works fine for months, right up until the one wrong row that mattered.