OCR models for tables and statements: structured OCR with provenance

Judge a table-and-statement OCR pipeline on one thing: can you point at any number in your database and pull up the pixels that produced it. That's the whole rule; everything else sits downstream of it. OCR gets called a solved image-to-text problem. Fine for a scanned letter. Wrong for a financial statement, a lab report, or a multi-page price schedule, where the system has to decide what belongs to the page, what order to read it, and whether the characters can be trusted.

The inputs are statements, schedules, price lists, lab results, tabular reports running many pages; the target is cells, headers, row hierarchy, merged regions, units, footnotes, continuation across page breaks. Working against you: faint rules, wrapped cells, repeated headers, a dozen ways to write a negative number. A vision-language model untangles layouts a plain OCR engine can't, and also, quietly, swaps an uncertain character for something that reads more naturally than the document does. Schema validity is necessary; what makes a result auditable is page, region, raw text, and a transformation history behind every field. Start with layout-aware OCR, table detection, and deterministic spreadsheet validation, add a generative model only where it earns its place, hard handwriting, irregular layouts, exception handling, never to replace stages already doing the job.

Store what the recognizer saw, not what looks right

A trustworthy pipeline returns source coordinates and raw spans, validates types and cross-field rules, and renders a review overlay for anything uncertain. The original file stays immutable, with a page and document identifier assigned before anything is processed. Record the chain of custody: file hash, page, capture device; original dimensions and orientation; preprocessing version; OCR or VLM model and runtime; raw text, reading order, and confidence; the normalized field with its validation and reviewer change; and resulting cost. Never let normalization overwrite transcription: an ambiguous O or zero, decimal, minus sign, unit, or diacritic gets stored as two things, what the recognizer saw and what the rule proposed. Measure the right target too, cell text, row-column association, header hierarchy, numerical consistency, table reconstruction, plus exact-match checks on identifiers, dates, and amounts. Error rate alone scores a missing comma and a wrong account number the same; keep table-geometry scoring separate from plain transcription accuracy.

The failure that doesn't look like a failure

The failure that bites is correctly recognized digits attached to the wrong row, column, period, or unit; nothing about it looks broken. Keeping only the corrected value is the real trap: later, nobody can tell it was verified, and because language models continue plausible patterns, a damaged total or part number gets completed into a familiar-looking alternative instead of failing loudly. Build an abstention path: low-resolution regions, glare, unreadable handwriting, and totals that don't add up should raise an exception, and pair poorly-calibrated confidence scores with image-quality checks and cross-field arithmetic. Provenance turns review into verification, not retyping: store the page and bounding polygon per field and show the crop beside the value. A VLM still earns a place linking a label to a distant value, but keep observation and interpretation separate and gate every write with schema validation outside the model: text in a document is data, not permission to redirect the extraction or call a tool. Multi-page work needs page order, repeated-header identity, and deterministic splits with an explicit continuation rule, or a loose stack of pages gets merged into one statement. Version the model, preprocessing, and schema together, and replay a stratified set, clean pages, common noise, critical fields, edge cases, whenever one changes. Local deployment keeps documents under your control and turns cheap at steady volume, against storage and accelerator cost; a hosted API or frontier model buys elasticity, against retention, residency, and a per-page bill; hybrid routing works when the privacy line is genuinely hard.

None of this is free, and I'll be honest: on a low-stakes internal report nobody will ever audit, the overlay and the coordinate storage are more plumbing than the document deserves, and a plain OCR pass with a human skim is probably the right call.