Vision models for document vision: an evaluation set for visual reasoning

A scanned invoice lands in the pipeline, gets resized and tiled before the model touches a raw pixel, and the total it hands back is copied from the wrong row. It feels concrete because you can see it, but the model gets a cropped, token-budgeted stand-in, and a fluent answer can be grounded, half-grounded, or built on a detail that never survived resizing.

Start with the parser, not the model

For document work, forms, invoices, reports, mixed-layout PDFs, feed the model page images with OCR text, page numbers, and a target schema, then score field, table, and source-location accuracy without losing qualifiers. The bigger decision happens before a vision model gets involved: run OCR and deterministic parsing first, escalate only when layout or language make rules too costly to maintain. A vision model earns its place. It doesn't start there. And the test set needs blur, glare, occlusion, tiny text, and bad crops, not clean scans that flatter a demo.

The paper trail you keep per call

Label which regions of the page carry the answer, run controlled perturbations, blind the outputs before scoring, and keep unsupported claims separate from plain omissions. Preserve the original and log every transformation before the call:

model ID, runtime, prompt revision
original dimensions, crop, resize, orientation
OCR or metadata supplied alongside pixels
visual, input, output token counts
first-response time, total latency, peak memory
grounded pass, unsupported claim, abstain, repair

The same discipline holds whether you're driving a hosted API through the Gemini CLI or running local weights through Ollama: log what it actually used, not the default you assumed. Local setups also need the projector, quantization, and context length logged, since a mismatched projector fails quietly while the server keeps talking.

Score the claim, not just the vibe

Judge the gate on field accuracy, table structure, source coordinates, abstention rate, and page-level review time:

• an omission gets flagged for human review
• an invented serial number, UI control, or trend produces a wrong action
• an unanswerable document belongs in the set, and admitting there's not enough evidence counts as a pass

Require a region, page, or visible feature behind every conclusion, and show it to the reviewer instead of hiding it in a log. Keep observation separate from inference: seeing an indicator isn't the same claim as a failed connection. With multiple images, say whether they're the same object or a sequence, and test them reshuffled: a model repeating the story after reshuffling isn't reading it. Crop faces and addresses before storage, and set retention separately for originals, OCR text, and debug captures: a local model removes the network hop, not the disk.

Before you promote anything, run it against reordered frames, a stack of unanswerable documents, and whatever blur your scanners produce, then compare the round trip at p50 and p95, not tokens per second alone.