OCR models for tables and statements: tables and key-value association

Someone, somewhere, has already signed off on a reconciliation that was wrong because a total from page four got attached to the header from page three. Nobody mistyped a digit. The OCR read every character correctly, it just landed the result in the wrong place, and the spreadsheet looked clean enough that nobody checked the scan again. That's the failure this kind of pipeline exists to prevent, and it's harder than reading characters off an image.

What a table actually asks you to prove

OCR gets sold as a solved problem: point a model at an image, get text back. For a financial statement, that's barely the start. The system must decide what belongs on the page, what order to read it in, whether the characters are trustworthy, and how each value maps to evidence. A table needs cells, headers, row hierarchy, merged regions, units, footnotes, and the fact that page nine continues page eight, not prose. The scans that make this hard are ordinary: faint rules, wrapped cells, repeated headers, negative numbers in parentheses, dense small type, and a vision model can help untangle that layout, or invent plausible text instead.

Preserve the associations before mapping anything into a business field, not the reverse. Start with layout-aware OCR, table detection, and deterministic validation, and let a generative model in only where it earns the spot: hard handwriting, irregular layout, a distant field association, an exception no rule covers. It doesn't replace the cheap stage that already works.

The paper trail every field needs

A fair test for a pipeline: can it show, for every extracted value, the bounding region it came from, the header path that names it, the row it belongs to, its unit, and an arithmetic check that row and column totals agree. Keep the source file immutable and assign a page and document identifier first.

Never let normalization overwrite what was read: an O and a 0, an I and a 1, a decimal point, a minus sign, a unit, a diacritic, store what the recognizer saw and what the rule decided, separately. Character or word error rate alone isn't enough: it treats a dropped comma and a wrong account number as the same mistake. Add exact-match scoring for identifiers, dates, amounts, and units, and score table geometry apart from transcription.

Fluent isn't the same as correct

The failure to watch for is correctly recognized digits landing on the wrong row, column, period, or unit. Nothing about that looks broken. Ask a model for a final JSON object directly and you lose the trail needed to notice a swapped field: a model just continues the plausible pattern, so a damaged part number or total can get completed into something that looks right and isn't.

Build a path for the system to say it doesn't know: low-resolution regions, clipped pages, glare, unreadable handwriting, and contradictory totals should raise an exception, not a guess. Confidence scores are model-specific and poorly calibrated, so pair them with image-quality checks, format validators, cross-field arithmetic, and disagreement between recognition passes.

Provenance turns review from retyping into verifying. Store the page number and bounding polygon for every field that matters, then show the crop beside the value. A reviewer settles an ambiguous digit in seconds when the region is visible. Without coordinates, the same check means searching a whole page, or digging out the original document.

What to let the model touch, and where you run it

A vision-language model earns its keep: a label tied to a value three columns away, a table split across a page break, two totals that need explaining. Keep observation and interpretation separate, and route any database write or tool call through schema and business rules outside the model. Text inside a document is data it reads, not an instruction it obeys, and a line telling it to skip validation isn't a reason to.

Multi-page work needs the same discipline: preserve page order and repeated-header identity, split documents deterministically, attach identifiers, and state whether a table may continue across a page boundary, rather than trusting the model to merge statements. The durable rule: structure and provenance first, normalization second, with model, preprocessing, dictionaries, schema, and validation versioned together, so a change forces a replay against a stratified batch of clean pages, noise, awkward layouts, and the examples built to break it.

Local deployment keeps documents under your own roof and gets cheaper the more steady volume you push through it, once you count storage, backup, accelerator time, and review headcount. A hosted API or a frontier vision model buys elasticity and a higher exception ceiling, at the cost of upload time, retention, residency, and a per-page or per-token bill. Hybrid routing works when privacy rules are unambiguous and escalation is visible.

I'd rather ship something slower that asks a person to glance at a crop than something that hands back a tidy JSON object I can't audit. That's the tradeoff, made on purpose: give up the one-shot answer and the demo where the model just reads the statement and hands you clean numbers, in exchange for a value you can stand behind when someone signs off on it.