Vision models for document vision: prompts grounded in visible evidence
Document-extraction pipelines have shipped the wrong invoice number to a payment run because it sat one row above the right one and looked close enough to pass review. Nobody flagged it. The number was real, just attached to the wrong invoice. That's the failure a vision-language model produces on a bare "read this and tell me what it says" brief.
The page looks concrete to the reviewer. What the model gets is cropped, resized, tiled, squeezed into a token budget. A fluent answer can be fully grounded, partly grounded, or built on a detail downsampled away, and nothing says which.
The picture the model never sees
Document work means scanned forms, invoices, reports, mixed-layout PDFs: ordinary to a person, not to a model reading a compressed raster of it. The job is pulling text, fields, tables, relationships, and source location without dropping a qualifier. Feed the model page images, OCR text, page numbers, and the schema, narrower than "understand this image." Narrower is the point: it makes failure measurable, not a vibe check.
The decision that matters happens before any prompt: the model reports what's visible before extending that with domain assumptions. Start with OCR plus deterministic parsing, and bring in a vision-language model only once that hits ambiguous layout, a relationship fixed rules can't express, or a language quirk custom code can't handle economically.
Write down more than the model's name
Ask for four things, in this order: what's observable, where it is, how confident the model is or whether it can answer at all, then a constrained conclusion. Keep the source image untouched and log every transform applied before the model saw it. A useful record tracks more than which model answered:
- model ID, API or runtime, prompt revision
- original dimensions, crop, resize, orientation
- image count, ordering, frame timestamps
- OCR or metadata supplied beside the pixels
- token accounting for visual, input, output
- first-response time, total latency, peak memory
- outcome: grounded pass, unsupported claim, abstain, repair
Log the resolution value actually used, not the default you assumed. A local GGUF through Ollama needs the projector, quantization, context length, and preprocessing implementation logged too, since text weights from one release paired with a projector from another can fail without looking like failure, the server just keeps answering fluently about the wrong crop.
Score field accuracy, table structure, source coordinates, abstention, and review time. Keep unsupported claims and missing information in separate buckets: an omission gets caught at review, but an invented serial number, UI control, or trend slides into a wrong action nobody double-checks. Put unanswerable examples in the test set, and score the model up for admitting it doesn't know.
Four kinds of claim, only one kind of proof
Not every sentence a vision model produces carries equal weight, and treating them as if it did is where the wrong-row failure comes from. "There's a red indicator in the upper right" is observation. "It belongs to the network panel" is association. "The connection has failed" is inference. "Restart the gateway" is action, the one with least room for error. Let a plausible observation walk straight into an action, and it restarts the wrong gateway.
Require a region, page, frame, or label behind every conclusion that matters, and show that provenance to the reviewer, not just in a log. Multiple images add an identity problem on top of the grounding one: label each input, and state whether they're the same object, page sequence, camera, or time window. Drop duplicate frames, they add no evidence. If order matters, attach timestamps and test a shuffle: a model telling the same story after the frames are scrambled isn't reading the sequence, it's pattern-matching.
Privacy, and the point where a bigger model stops helping
The rule that holds up in production: every conclusion points back to a visible region, label, frame, or supplied fact, and that belongs in preprocessing, routing, or validation code, not a paragraph of prompt text. Store the model and pipeline version with every result, so an upgrade can be replayed against the same images later.
Privacy applies on both sides of inference. Crop out rooms, faces, screens, addresses, and reflections unrelated to the task, and set retention separately for originals, thumbnails, OCR text, embeddings, prompts, and debug captures. Running the model locally cuts down what leaves the building; it does nothing for an unencrypted disk or an overbroad dashboard account.
For a hosted setup, through the Gemini CLI's multimodal mode, test upload time, regional routing, retention, rate limits, and provider errors. For a local setup it's cold-load time, projector memory, GPU placement, preprocessing time, heat, and concurrency. Measure the whole path at p50 and p95, tokens per second alone tells you nothing about what a reviewer waits for.
Stop tuning once the gate passes at a latency, cost, and abstention rate you can live with. A bigger model and more pixels aren't automatically safer, they can surface more private detail than needed and make it easier to answer past the evidence.
What I haven't solved is the provenance display itself. A bounding box the model draws around "the evidence for this field" comes from the same weights that might be hallucinating the field, so it isn't independent proof, it's the model grading its own homework, and a tidy panel makes it look more trustworthy than it is. No clean fix. Just a habit of not trusting a box until something outside the model agrees.