Vision models for document vision: resolution and visual-token budgets
A scanned invoice goes into a vision model. You ask for the total. It answers with total confidence, and the number is real, just pulled from the row above the one you meant, because that's the row that survived the crop before the image ever reached the model. The model has no idea the row above existed too. It only saw what you gave it.
That's the whole problem with document vision in one sentence: the picture you're looking at and the picture the model receives are not the same object. Somewhere between your file and the forward pass, the page got resized, tiled, cropped, or squeezed into a fixed token budget, and whatever process did that is now quietly deciding what the model is and isn't allowed to see.
Decide what has to survive the resize
The workload here isn't "look at a picture and tell me what's in it." It's scanned forms, invoices, reports, and mixed-layout PDFs, and the job is narrow: extract text, fields, tables, the relationships between them, and where each answer came from on the page. Feed the model page images alongside OCR text, page numbers, and the schema you want filled in. That's a much smaller ask than "understand this image," and it's smaller on purpose, because a bounded question is the only kind you can actually evaluate, compare across model choices, and hold to a failure standard.
The decision that matters most is which pixels earn their place. Not the camera's maximum output, the smallest region of the page that contains the evidence the task needs to resolve. Start with OCR and deterministic parsing, the boring non-generative baseline, and reach for a vision-language model only once that baseline hits layout ambiguity, cross-field relationships, or language a fixed rule can't handle without turning into its own maintenance project.
Once a VLM has earned its keep, the experiment is a sweep: image dimensions, crop strategies, task accuracy, visual-token counts, first-response time, total cost, all logged against the same document set. Keep the original asset untouched and write down every transformation you applied before the call, because the eval record has to hold more than a model name:
model ID, API/runtime, prompt revision
original dimensions, crop, resize, orientation
image count, ordering, frame timestamps
OCR or metadata supplied beside pixels
visual/input/output token accounting
first response, total latency, peak memory
grounded pass, unsupported claim, abstain, repairIf the provider exposes an image-detail or resolution setting, log what it actually resolved to, not the default you assume it used. Running local, through something like Ollama's vision models, write down the vision projector, the quantization, the context length, and how images get preprocessed before they hit the model. Text weights from one release paired with a projector from another can fail in a way that's hard to spot: the server keeps returning fluent, well-formed language, it's just language that quietly stopped being about the image several layers back.
Score the result against a gate: field accuracy, table structure, source coordinates for each answer, abstention rate, and how long a human spends reviewing the page. Treat unsupported claims as a different failure class than missing information:
- an omission gets caught in review, which is annoying but safe
- an invented serial number, UI control, or total gets acted on, which is not
Put a few genuinely unanswerable documents in the set and reward the model for saying so.
The wrong-row failure and how to catch it
The most common way this goes wrong is boring: a field value that's visually plausible, formatted correctly, and pulled from the wrong row, wrong page, or wrong section of the document. It reads fine. That's what makes it dangerous, because a reviewer skimming for plausibility won't catch it. There are two specific ways to lose the pixels that would have caught it first: downscale a whole page until the text you actually need is a smear, or do the opposite and hand the model a full, uncropped scan to answer a question about one paragraph, diluting the evidence across everything else on the page. Require a region, page number, or visible label behind every answer that matters, and show that provenance to the reviewer instead of burying it in a log nobody reads.
It helps to keep the model's reasoning in layers instead of one blended paragraph. "There's a stamped date in the upper right" is observation. "It belongs to the approval block" is association. "The document was approved on that date" is inference. "File it as approved" is an action, and it's the one layer that should never fire off the back of a plausible-sounding observation with nothing checking the chain underneath it.
Multiple images multiply this risk instead of averaging it out. Label each input, state explicitly whether they're the same document, sequential pages, or unrelated attachments, and drop duplicate frames that cost tokens without adding evidence. If order matters, attach timestamps or page numbers and run a shuffle test: a model that tells the same story after you reorder the images isn't using the sequence, it's pattern-matching on content alone.
The cost shows up in two different places depending on where the model runs. Hosted, you're paying in upload time, regional routing, retention terms you didn't write, rate limits, and the provider's own error rate. Local, through something like the multimodal path in Gemini CLI or a self-hosted VLM, you're paying in cold-load time, projector memory, GPU placement, preprocessing time, heat, and concurrency. Measure the whole path at p50 and p95, not tokens per second alone, because that number looks good in a slide and says nothing about what a reviewer waits on.
Privacy sits in the same budget. Crop out rooms, faces, screens, and addresses that don't belong in the answer, and set retention separately for originals, thumbnails, OCR text, and debug captures. A local model keeps the bytes off someone else's server. It does nothing about an unencrypted disk or a dashboard account with too many people logged into it.
The rule I keep: pixels are a budget, not a virtue. Spend them on the region that has to be right and cap everything else, because a bigger image and a bigger model don't make a wrong row less confident, they just make the wrong answer higher resolution.