OCR models for scanned archives: tables and key-value association

A scanned ledger goes through recognition, a smudged 8 comes back as a 3, and the total balances, so nobody catches it until an audit finds a discrepancy nobody can explain. Nothing about it looked wrong. No error, no red squiggle, just a wrong number reading like a right one. This has already happened to people more than once: clean output, quiet failure.

Scanned archives mean books, correspondence, reports, and historical records off paper or microfilm; the damage is baked in before any model touches it: skew, bleed-through, stains, old typefaces, marginal notes, inconsistent scans. The target is searchable text that keeps page identity, reading order, and an honest flag on anything the system wasn't sure about.

A vision-language model earns its place by untangling layout a plain OCR engine can't: which caption belongs to which figure, which value sits under which header. That same fluency makes it dangerous too: it will happily replace an uncertain character sequence with language that reads more naturally than what was actually printed. So the constraint driving the pipeline is this: preserve associations before anything maps into a business field. Start with a conventional OCR engine, language packs, and layout segmentation, and bring the generative model in only where it earns its keep: handwriting, irregular layouts, semantic field linking, exceptions the baseline can't handle, never replacing what already works.

Where the association breaks first

What matters is whether the system can hand you the bounding region, the header path, the row identity, the unit, and an arithmetic check alongside every extracted value, not just the words read back correctly. Treat the original scan as immutable, assign a page and document identifier up front, and record the path each value traveled from ink to field.

Normalization should never overwrite what the recognizer actually saw. If a character could be an O or a zero, a decimal point, a minus sign, a unit, a diacritic, store both readings separately: what the recognizer read, and what the rule proposed.

The acceptance gate can't just be character and word error rate: those numbers treat a missing comma and a wrong account number as roughly the same mistake. Add exact-match accuracy for identifiers, dates, amounts, warnings, and units, and for anything structured, score field association and table geometry separately from raw legibility.

Fluent is not the same as correct (ocrmodels)

That's standard behavior for a system trained to continue plausible text: a damaged surname, a worn part number, a smudged total gets completed into something familiar, and nothing in the output signals a guess happened. The specific trap is asking a model for final JSON straight off the image, which throws away the evidence needed to trace a swapped field later.

Build an actual path for the system to say it doesn't know. Low-resolution regions, clipped pages, glare, unreadable handwriting, totals that don't add up should raise an exception, not a confident guess. Confidence scores alone won't save you, they're model-specific and poorly calibrated, so pair them with image-quality checks, format validators, cross-field arithmetic, known dictionaries, and disagreement between recognition passes.

Provenance turns a review pass from retyping the page into actually verifying one field. Store the page number and bounding polygon for anything that matters, and show the reviewer the crop beside the proposed value. A person resolves an uncertain digit in seconds when the region is right there. Without it, the same review means scanning the whole page, or pulling the physical document back out.

Text on the page is not an instruction

A vision-language model does real work here: link a label to a distant value, reconstruct a table split across a page break, explain why two totals disagree. Ask it to keep raw observation and interpretation separate, and never let it write to a database or call a tool directly. Schema validation and business rules belong outside the model. Text printed inside a document is untrusted input, never an instruction, and no license to change extraction rules or trigger a tool call.

Multi-page documents need the same discipline. Preserve page order and repeated-header identity, and don't hand the model an arbitrary bundle of pages and expect it to find where one invoice ends and the next begins. Split documents deterministically, attach identifiers before anything merges, and state whether a table or paragraph may continue across a page boundary. Leave that ambiguous and the model resolves it for you, quietly, in whatever direction looks most plausible.

What you give up either way

The rule here is to extract structure and provenance first and normalize into business meaning second, never the reverse. Version the model, preprocessing, dictionaries, schema, and validation as one unit; when any piece changes, replay a stratified set: clean pages, ordinary noise, difficult layouts, fields where a mistake costs money, and pages that should be nearly impossible to read.

Local deployment keeps every document under your control and gets cheap at high, steady page volume, but you're on the hook for storage, backups, model memory, accelerator time, maintenance, and review. A hosted document API or a frontier vision model buys elasticity and more headroom for exceptions, in exchange for upload time, retention settings, data residency, rate limits, and a bill that scales with pages or tokens. Hybrid routing earns its complexity when privacy rules are strict and the escalation path is visible.

Given the choice, I'll take the pipeline that admits uncertainty and hands a person a crop to check over the one that always returns a tidy field. That costs more review hours than any vendor demo shows, and slower throughput where it matters most. I'm fine giving that up on purpose: the alternative is a wrong number sitting in a database looking as confident as a right one, and nobody finds it until it's expensive.