OCR models for tables and statements: an OCR evaluation that predicts production

A folder of scanned bank statements and price lists doesn't arrive clean, doesn't arrive in one format, and doesn't care what your model scored on a benchmark. This material, financial statements, schedules, lab results, multi-page tabular reports, comes with faint table rules, wrapped cells, repeated headers, uneven indentation, negative numbers in parentheses, and small type. What has to survive the trip from image to structured data isn't just words: cells, headers, row hierarchy, merged regions, units, footnotes, tables spanning a page break.

Something has to decide what belongs on the page, what order to read it, and how each value ties back to the pixels behind it. A vision-language model helps here, tracing a label to a distant value or rebuilding a split table. It can also swap an ambiguous character for whatever looks natural, and that's the failure mode here.

What you log matters more than what you read

Weight errors by downstream consequence, by how often that document type shows up, not by how they look on a leaderboard. Build the test set from real capture conditions, annotate structure and text, and start with layout-aware OCR, table detection, and deterministic spreadsheet validation, cheap stages that already work. Let a generative or multimodal model earn a place only where it shows a measured advantage: bad handwriting, irregular layouts, a field tied to a distant label, or exceptions the deterministic stage kicks out.

Treat the source file as evidence: keep the original immutable, assign a page and document identifier, and log the chain from capture to reviewer sign-off.

source hash, page, capture device
dimensions, color, orientation
crop, dewarp, threshold, denoise versions
OCR/layout/VLM model, revision
language hints, dictionaries used
raw text, regions, reading order, confidence
normalized field, validation, reviewer change
latency, memory, energy or API cost

Never let normalization overwrite what the recognizer saw: an O that might be a 0, an I that might be a 1, a decimal, a minus sign, a unit, a diacritic, store the raw read and the business-rule guess separately. Don't lean on CER and WER alone either; they treat a dropped comma and a wrong account number as the same mistake. Add exact-match scoring for identifiers, dates, amounts, warnings, units, and score table geometry apart from raw transcription.

Fluent wrong answers are the actual enemy

The failure that costs money isn't garbled text: it's a correctly recognized digit in the wrong row, column, or period, paired with the wrong unit, looking as clean as everything else on the page. Score only clean pages, or publish one aggregate number dominated by easy body text, and you'll never catch it. Language models continue plausible patterns, so a damaged surname, part number, or total doesn't come out as noise, it comes out fluent. That's the expensive kind.

Build in abstention: some conditions should raise an exception, not a guess.

• low-resolution or clipped regions
• glare across the relevant cell
• handwriting nobody could read with confidence
• totals that contradict each other

Confidence scores are model-specific, often poorly calibrated. Cross them with image-quality checks, format validators, cross-field arithmetic, dictionaries, and disagreement between recognition passes.

Store the page number and bounding polygon for every field that matters, and show the crop beside the proposed value: a reviewer clears an uncertain digit in seconds with the region visible. Skip the coordinates and that check means hunting a whole page, or pulling the document back out.

Keep the model on a leash

A vision-language model earns its keep on reasoning: tying a label to a distant value, rebuilding a table split by a page break, explaining why two totals disagree. Keep its observation and interpretation in separate fields. Any database write still goes through schema validation and business rules outside the model. Text on a page is data the model reads, never an instruction it obeys, and a footer trick isn't permission to change extraction or call a tool.

Multi-page documents need the same discipline: split them deterministically before a model sees them, attach identifiers, and state whether a table may continue across a boundary. Hand it an unstructured pile of pages, and it will merge two statements into one, or drop a repeated header it decided was redundant.

Local deployment keeps documents under your control and gets cheaper at steady, high volume, once you count storage, backup, model memory, accelerator time, and maintenance nobody budgets for. Hosted APIs or frontier models buy elasticity and a higher ceiling for strange documents, at the cost of upload time, retention, residency, rate limits, and a per-page bill. Hybrid routing earns its complexity when privacy rules are strict and escalation stays visible. I wouldn't spend engineering time chasing a marginal CER gain on clean, well-lit invoices; that batch was never where the money leaked.

The one rule I would not drop: never let a changed model, prompt, or preprocessing step into production without replaying it against a stratified set of real failures first, clean pages included.