Vision models for document vision: preprocessing before the vision model
Preprocessing beats a bigger model on most document-extraction problems. That claim needs a qualifier: only for mess a script can fix, crooked scans, wrong crops, blown contrast, a giant PDF. A vision-language model buys nothing there but a bigger bill.
Document vision means scanned forms, invoices, reports, and mixed-layout PDFs: pulling fields, tables, and locations from a page image, alongside OCR text and a target schema. The model never sees the page as you do, only whatever crop and resolution the pipeline handed it, cut to a token budget nobody checked, so a fluent answer can be grounded, partly grounded, or built around a vanished detail.
The log you wish you'd kept
Start from the boring baseline, OCR plus deterministic parsing, before a vision-language model touches anything. Bring the VLM in only where that baseline fails: ambiguous layout, field relationships, or language fixed rules can't cover cheaply. Preprocessing's job is stripping irrelevant variation without touching the evidence. That's the whole decision rule.
Version the pipeline: run raw, normalized, cropped, and OCR-augmented inputs as separate conditions, original untouched behind every transform. Log runtime, prompt revision, crop, orientation, image count, OCR or metadata supplied, tokens, latency, and outcome: grounded, unsupported, abstained, repaired. Judge it against field accuracy, table structure, source coordinates, and review time per page; score unanswerable cases as wins, not misses.
The model never sees the page the way you do.
Log the resolution setting used, not the default assumed. Running local, log the projector, quantization, context length, and whatever code handles preprocessing, a layer hosted Gemini CLI calls skip and local Ollama setups can't. Mismatched weights, text from one build and a projector from another, are a nasty failure: the server keeps answering fluently while it guesses at everything visual.
The row it borrowed the answer from
The failure that bites: a plausible field value pulled from the wrong row, page, or section, attached to the wrong record and waved through by a tired reviewer. Score unsupported claims apart from omissions: a missing field gets flagged, an invented serial number triggers an unchecked action. A sibling trap: over-sharpening that reads clearer to a human eye while it invents a character or hides a defect.
Split every conclusion into observation, association, inference, and action: "red indicator, upper right" is observation, "the network panel's" is association, "connection failed" is inference, "restart the gateway" is action, needing the most evidence of the four. Several images add identity risk: label each one, say whether they're the same object, sequence, or time, drop duplicates, and if order matters, timestamp and test with a shuffled control.
The rule that lives in code
Put the discipline, keep the original, make every transform reproducible, into pipeline code, not prompt wording. Store model and pipeline version with every result, for replay later.
Privacy sits on both sides of inference: crop out rooms, faces, screens, addresses, and reflections you don't need, and set separate retention for originals, OCR text, embeddings, and debug captures. Local cuts what leaves the building; it won't encrypt your disk.
Measure the whole path, not tokens per second: hosted upload and routing, local cold load and projector memory, both at p50 and p95. Stop tuning once the gate passes at acceptable latency and cost. A bigger model or more pixels isn't automatically safer, it can expose more of the document and invite answers past what's on the page.
So before reaching for a bigger model, pull your last batch of failures: did the wrong answer come from a detail the image never showed, or one it showed and got wrong anyway. That's the difference between a fix in the crop and a fix in the weights.