OCR models for tables and statements: capture quality before recognition

Here's the rule I use before any OCR pipeline touches a statement or a table: if the smallest character on the page doesn't have enough clean pixels behind it, no model downstream will save you, so fix the capture or throw the page out. Everything else here is just me explaining why I hold that line so hard.

Financial statements, fee schedules, price lists, lab panels, multi-page tabular reports: these documents ask you to reconstruct cells, header hierarchy, merged regions, units, footnotes, plus whatever continues onto the next page. Faint rules, wrapped cells, repeated headers, indentation that carries meaning, negative numbers in parentheses, dense small type crammed into a scan. None of that is exotic. It's just what statements look like, and it's exactly what a blurry or skewed capture destroys first: the pixels carrying a decimal point or a minus sign were already the smallest ones on the page.

Reject the page before it gets near a model

My baseline is boring on purpose: layout-aware OCR with table detection, feeding deterministic spreadsheet validation. A vision-language model gets invited in only where it earns its keep: difficult handwriting, an irregular layout, associating a label with a value three columns away, or an exception. It doesn't get to replace the deterministic stage that already works.

Before any of that runs, the capture itself needs a gate: checks for blur, glare, clipping, skew, resolution, page completeness. Fail it and you reject or reacquire, before the page reaches a recognizer. The original file stays immutable, tagged with a page and document identifier, and every downstream step gets logged against it: source hash, capture device, preprocessing version, model and runtime revision, language hints, and the raw output, kept separate from the normalized field and any reviewer change. Latency, memory, cost too: someone eventually asks why a batch got slow, and the answer should live in the log, not in your head.

One habit that pays for itself: never let normalization overwrite what the recognizer actually saw. An ambiguous O versus 0, an I versus 1, a decimal point, a minus sign, a unit, a diacritic: when the raw guess and the cleaned-up business-rule guess disagree, store both. Collapse them into one field and that invoice becomes unreadable evidence the moment someone audits it.

The failure that looks fine is the one to worry about

The failure mode I lose sleep over isn't garbled text. It's correctly formed digits sitting in the wrong row, wrong column, wrong period, or wrong unit: output that passes a glance because nothing about it looks broken. The specific trap is handing an unreadable crop to a big model and getting back a fluent, plausible completion. Language models are trained to continue a pattern convincingly, so a damaged surname or part number gets finished into something that reads better than the source ever did.

Character error rate and word error rate are fine metrics, and also the wrong ones to trust here: they score a dropped comma about the same as a wrong account number. I want exact-match checks on the fields that matter (identifiers, dates, amounts, warnings, units), plus separate scores for field association and table geometry.

The other half of the job is giving the system permission to say no. Low-resolution regions, clipped pages, glare, handwriting nobody can read, totals that don't reconcile: those should raise an exception, not a confident guess. Confidence scores are poorly calibrated and vary by model, so I lean on them alongside image-quality checks, format validators, cross-field arithmetic, dictionaries, disagreement between recognition passes. When a field gets flagged, show the reviewer the actual crop next to the proposed value, tied to a page number and bounding polygon. That turns review into verification instead of retyping a document from scratch.

Treat the model as a witness, not a clerk

A vision-language model is useful for connecting a label to a distant value, stitching a fragmented table back together, or explaining why two totals disagree. Ask it to observe, not decide. Anything that writes to a database or calls a tool needs schema validation outside the model: text printed inside a document is data the model reads, not an instruction it should obey, and a well-crafted statement footer is a plausible place to smuggle a prompt.

Multi-page work has its own version of this problem. Don't hand a model a loose bundle of pages and trust it to keep invoices and statements separate. Split documents deterministically, attach identifiers up front, and say explicitly whether a table or paragraph may continue across a page break. Let the model guess at boundaries instead, and two statements become one.

Pick where the pipeline runs, and mean it

Version the model, the preprocessing, the dictionaries, the schemas, and the validation rules as one unit. When any piece changes, replay it against a stratified set: clean pages, ordinary noise, hard layouts, the fields you can't afford to get wrong, a few impossible examples that should trigger abstention instead of a guess.

I'd run this locally when volume is steady and documents can't leave the building: at real scale, storage, backup, model memory, accelerator time, and ongoing maintenance beat a per-page bill. A hosted document API or a frontier vision model buys elasticity and a higher ceiling on the weird cases, at the price of upload time, retention settings, data residency questions, rate limits, and cost that scales with every page. Hybrid routing earns its complexity only when the privacy rule is non-negotiable and the escalation path stays visible to a human.

That's the tradeoff I accept on purpose: slower, less elastic handling of the ugly long tail, in exchange for never having to explain why a client's statement left the building.