OCR models for tables and statements: layout and reading order
None of that changes the fact that a model can read every character right and still hand you the wrong number, because it filed them under the wrong row. Score a pipeline near-perfect on individual glyphs and it still misfiles subtotals into the wrong account.
The grid is the message, not the ink
Financial statements, schedules, price lists, lab results, multi-page tabular reports. What matters: cells, headers, row hierarchy, merged regions, units, footnotes, and whether a table spanning a page break still counts as one. The usual damage: faint rules, wrapped cells, repeating headers, encoded indentation, parenthetical negatives, dense type. A vision-language model can help read an ambiguous layout, or replace an uncertain character with fluent, wrong text. Start with layout-aware OCR, table detection, deterministic spreadsheet validation. Bring in a generative model only where it earns its keep: hard handwriting, irregular layouts, distant label-value pairs, exceptions the deterministic stages refuse to guess at.
Test the shape before you test the words
Build a test set from layouts that break things: multi-column pages, sidebars, repeating headers and footers, page-spanning tables, and score structure apart from text. Keep the source file immutable, stamped with a page and document id first. Log everything: file hash, page, capture device; dimensions, color, orientation; crop, dewarp, threshold, denoise version; OCR, layout, or VLM model and runtime revision; language hints and dictionaries; raw text, regions, reading order, confidence; normalized field, validation, reviewer change; latency, memory, energy, or API cost. Never let normalization overwrite transcription. When a character is ambiguous, an O or zero, a decimal point, minus sign, unit, or diacritic, store what the recognizer saw and what the rule guessed, separately.
A comma and a wrong account number don't grade the same
The acceptance gate runs on cell text, row-column association, header hierarchy, numerical consistency, and table reconstruction quality. Character and word error rate are cheap but weight a dropped comma the same as a transposed account-number digit, a very different mistake. Add exact-match checks for identifiers, dates, amounts, warnings, units, and score field association and table geometry apart from transcription.
The failure that looks fine until someone checks
The trouble is where correctly recognized digits land: wrong row, column, period, or unit, output that passes every surface check because nothing looks wrong. The trap here: flattening a page into one string and asking a language model to reconstruct the geometry it just threw away. Trained to continue plausible patterns, it completes a damaged surname, part number, or total into something merely plausible.
Build an abstention path, not a threshold bolted on after. Low-resolution regions, clipped pages, glare, unreadable handwriting, contradictory totals: raise an exception, not a guess. Confidence scores are model-specific and often miscalibrated; pair them with image-quality checks, format validators, cross-field arithmetic, known dictionaries, and disagreement between passes.
Store the page number and bounding box for fields that matter, and show the crop beside the extracted value. That turns review from retyping into verifying.
A reviewer resolves an uncertain digit in seconds when the crop sits next to it; without coordinates, review means re-reading the whole page.
Read the page, don't take orders from it
A capable OCR-VLM can tie a label to a distant value. It can stitch a fragmented table back together, or explain mismatched totals, but keep observation separate from interpretation. Tool or database writes need schema validation and business rules outside the model. Text on a page is untrusted input: it can't rewrite instructions or trigger a tool call. Multi-page work needs the same discipline: preserve page order and repeated-header identity, never merge an arbitrary page bundle into one statement, split deterministically, and state whether a table may cross a page boundary.
Local deployment keeps documents under your control and gets cheaper at steady volume, once you count storage, backup, model memory, accelerator time, and maintenance. Hosted APIs and frontier vision models buy elasticity and a higher ceiling for hard exceptions, at the cost of upload time, retention, residency, rate limits, and a per-page or per-token bill. Hybrid routing earns its complexity when privacy rules are strict and escalation stays visible.
Version the model, preprocessing, dictionaries, schema, and validation together. When any changes, replay a stratified set: clean pages, ordinary noise, hard layouts, fields you can't afford to get wrong, a few pages you expect to fail.
Never let a system report a number it can't point to on the page.