Vision models for document vision: structured output from images

Run OCR and deterministic parsing first. Reach for a vision-language model only when the layout, the ambiguity, or the relationships in a document beat what fixed rules can do economically. That's the rule I use, and the rest of this is why I trust it enough to wire into something that writes to a database.

Document vision, in the boring sense I mean it, is scanned forms, invoices, reports, and mixed-layout PDFs. The job is pulling text, fields, tables, and the relationships between them, plus knowing where each value came from, without softening qualifiers the source document contains. Feed the model page images next to OCR text, page numbers, and the schema you want back. That's a narrower ask than "look at this and tell me what it says," and narrower is what makes the rest of the pipeline measurable.

Here's the part people skip: a schema tells you the shape of an answer, never its source. A model can validate cleanly against your JSON schema and still be wrong, pulling a number from the row above, the previous page, or a half-remembered section header. Validation catches malformed output. It doesn't catch wrong output dressed up correctly.

What I log besides the model name

I log more than a model name and a score, because the failure modes live in the gaps between them. A local vision-language model, the kind I've run through Ollama's vision models, can pair a text checkpoint from one build with a vision projector from another and keep answering, degraded, while the runtime reports nothing wrong. Each run's record carries the image handling alongside the output, not just the score:

record:
  model: id, runtime, prompt_revision
  image: orig_dims, crop, resize, orientation
  sequence: count, order, frame_timestamps
  context: ocr_text, page_numbers, schema_version
  tokens: visual, input, output
  timing: first_token, total_latency, peak_memory
  outcome: grounded | unsupported | abstain | repaired

If the provider exposes an image-detail or resolution knob, the kind you'd tune through Gemini CLI's multimodal input path, log the value it used rather than the default you assumed. Score unsupported claims apart from missing fields: a blank field sends a human to look, but an invented serial number, control, trend, or product attribute can trigger an action unchecked. Build genuinely unanswerable cases into the test set, and reward the model for saying so instead of guessing toward a schema-shaped answer.

Wrong row, right shape

The failure that bites is a plausible value pulled from the wrong row, the wrong page, or the wrong document entirely. It survives review because it looks like what a tired reviewer expects, and a schema-valid hallucination walks straight into automation unnoticed. The fix is boring: require a page, region, or visible feature behind every consequential field, and show that provenance to the reviewer instead of burying it in a log nobody opens.

Keep prompts honest about what kind of claim they're making. "There's a red indicator, upper right" is an observation. "It belongs to the network panel" is association. "The connection has failed" is inference. "Restart the gateway" is an action. Each step needs its own evidence, and a model should never leap from the first straight to the last.

Multiple images add an identity problem too. State plainly whether frames show the same object, the same page run, the same camera. Drop duplicates that just burn tokens. If order matters, shuffle it in testing: a model that tells the same causal story after you scramble the sequence isn't reading the sequence, it's pattern-matching around it.

I take the tradeoff this makes on purpose. An OCR-first, provenance-gated pipeline is slower to build and slower to answer than pointing a large model at a folder of PDFs and trusting whatever JSON it hands back. I give up the one-call simplicity and some of the raw throughput a demo would show off. What I get instead is a system that fails by shrugging rather than by lying convincingly, and for anything that writes to a database unattended, that's the only trade I'd sign my name to.