OCR models for invoices and receipts: tables and key-value association

A thermal receipt comes in with the total printed three times: once above the line items, once near the tip line, once more on a duplicate stapled behind it. One of those numbers is the one your ledger needs. The OCR engine has no opinion about which, because to a character recognizer they're all just digits on a page.

That's the ordinary case. Supplier invoices, retail receipts, credit notes, photographed expense slips arrive crumpled, faded from thermal paper gone bad, shadowed, angled, logo-covered. The job is pulling vendor, dates, totals, tax, currency, and line items, each tied to a coordinate on the source page, into a schema someone downstream can validate.

Getting the characters right is the easy part now. Proving a number is the subtotal and not the balance-forward, that a string is the vendor's tax ID and not a reference code nearby, is the actual job. This is where a vision-language model hurts you without you noticing: fed a damaged run of characters, it doesn't fail loudly. It completes the pattern into something that reads more fluently than what was printed, and hands you a wrong answer with the confidence of a right one.

The pipeline I'd build starts boring on purpose: plain OCR, vendor templates, anchors, arithmetic checks that a subtotal plus tax equals the total. A generative or VLM stage earns a seat only where it beats that: ugly handwriting, layouts that break the templates, a label far from its value, or the exception queue once the deterministic stages are done. It doesn't replace what already works just because it's newer.

The paper trail that has to outlive the extraction

A test I'd hold any system to: can it produce, alongside every value, the bounding region it came from, the header path above it, its row identity, its unit, and the arithmetic check that validated it. A field that can't answer those isn't extracted, it's guessed with good formatting.

Treat the source file as immutable and stamp it with a page and document identifier before anything else touches it. From there you're accumulating a record, not overwriting one:

• what the file looked like before anything touched it: hash, page, capture device, dimensions, orientation
• what preprocessing did: crop, dewarp, threshold, denoise, and the version of each step
• what recognized it: OCR, layout, or VLM model, runtime revision, language hints, dictionaries used
• what came out: raw text, regions, reading order, per-token confidence
• what changed afterward: normalized field, validation result, reviewer edit, plus latency, memory, or cost

Never let normalization overwrite the transcription it's correcting. When a character is ambiguous, an O that could be a zero, a decimal point, a minus sign, a unit, a diacritic, store what the recognizer saw next to what the rule proposes. Collapse those into one field and a wrong guess becomes indistinguishable from a confirmed read months later.

Measure the pipeline on field and line-item accuracy, arithmetic reconciliation, duplicate detection, and how often it punts to a human. Character and word error rate are fine as secondary signals, but they'll call a system great while it drops a digit from an account number: to CER, a missing comma and a wrong account number look about the same. Add exact-match scoring for identifiers, dates, amounts, currency, and warnings raised. For table-shaped fields, score geometry, whether a cell landed in the right row and column, separately from character accuracy. Different failure modes; averaging them hides both.

Where the model earns its keep, and where it quietly lies

The failure that should worry you isn't garbled output. It's a syntactically perfect amount, correct format, correct currency symbol, copied from the subtotal line instead of the total, or from the invoice next to it in a scanned batch. Nothing about it looks broken. Asking a model for final JSON straight off the image makes it worse: you lose the evidence needed to work out which field got swapped, and why.

Build an abstention path and mean it. Low-res regions, clipped pages, glare on the total line, unreadable handwriting, totals that don't reconcile: these raise an exception, not a dressed-up guess. Confidence scores are model-specific and often miscalibrated; pair them with image-quality checks, format validators, cross-field arithmetic, known dictionaries, and disagreement between two independent passes.

Provenance turns a review queue from retyping into verification. Store the bounding polygon for every field that matters and show the reviewer the crop beside the value: a glance at a digit instead of hunting a full page, or worse, tracking down the physical receipt.

Used carefully, a VLM earns real work: linking a distant label to its value, reconstructing a table fragmented by a fold or a stain, explaining why two totals disagree. Ask it to keep observation and interpretation separate rather than blend them into one answer. Any write to a database or tool still goes through schema validation and business rules living outside the model. Text printed inside the document is data the model reads, never an instruction it follows, never a reason to call a tool it wasn't asked to call.

Across multi-page documents, preserve page order and repeated-header identity, and split batches deterministically before anything reaches the model rather than letting it decide where one invoice ends and the next begins. State explicitly whether a table or paragraph may continue across a page boundary.

Version the model, preprocessing, dictionaries, schema, and validation rules together, as one unit. When any changes, replay a stratified set: clean pages, common noise, ugly layouts, fields that matter, and a few documents you expect to fail outright.

I'd run this locally where volume is high and steady: it keeps documents under direct control, and the ongoing cost becomes storage, backup, accelerator time, and review labor, not a per-page bill. Hosted document APIs and frontier vision models buy elasticity and a wider ceiling on hard exceptions, at the cost of upload time, retention settings, residency questions, rate limits, a running per-page charge. Route the bulk locally, send the acknowledged-hard cases out, and only if that escalation path stays visible and audited, not a quiet fallback nobody's watching.

That's the tradeoff I'd take on purpose: a deterministic-first pipeline, slower on a novel layout, costing more engineering time up front on anchors and validators. In exchange I get evidence to point to when a number is wrong, and a review queue that never asks anyone to retype a receipt from scratch.