OCR models for invoices and receipts: an OCR evaluation that predicts production

Somewhere a reconciliation team spent weeks chasing a mismatched total back to one invoice: the recognizer had quietly copied the subtotal into the total field, and the check downstream passed because the copied number was syntactically perfect. Nobody caught it; nothing looked wrong. That is the whole problem with OCR on invoices and receipts: the failures that cost money do not look like failures.

Treat OCR as a solved image-to-text conversion and you build the wrong system. A pipeline decides what belongs on the page, in what order, whether the characters are trustworthy, and how each number maps to something checkable, or it drifts quietly for months.

The inputs are supplier invoices, retail receipts, credit notes, and photographed expense reports: crumpling, fading thermal paper, shadows, perspective, logos, totals repeated across sections. The output is vendor, dates, totals, taxes, currency, line items, and the coordinates each value came from. A vision-language model helps with ambiguous layout, but can replace an uncertain character with plausible-sounding text that nobody flags.

Weight errors by downstream cost and document frequency, not by how clean the score looks. Start with OCR, vendor templates, anchors, and arithmetic validation, cheap and already handling most volume. A generative model earns a place only where it shows a measured advantage, not by replacing stages that work.

The log you wish you had after the fact

Build the test set on real capture conditions, not clean scans photographed under office light. Annotate text and structure by hand, then report CER and WER alongside two numbers most teams skip: critical-field accuracy and the abstention rate, how often it correctly flags uncertainty.

Keep the original file immutable and stamp it with a page and document identifier before anything touches it. Lose that trace and regression testing is vibes.

Never let normalization overwrite what was transcribed: if a character could be O or 0, a decimal, a unit, or a diacritic, store what the recognizer saw and what the rule proposed as two separate values. The acceptance gate is field and line-item accuracy, reconciliation, duplicate detection, and exception rate; CER and WER weight a missing comma the same as a wrong account number. Add exact-match metrics for critical identifiers, dates, and amounts, and score field association apart from transcription.

A wrong number that looks exactly right

The failure worth designing around is not garbled text. It is a syntactically valid amount lifted from the subtotal, the balance-forward line, or the line next to the one that mattered. Nothing looks broken, which is why testing only clean pages, or one aggregate score that easy text drags upward, hides the fields you actually care about.

Language models, generative OCR included, are trained to continue plausible patterns. Feed one a damaged surname or a total and it often completes it into something that looks normal rather than admit it cannot read the original: exactly the instinct you did not want.

Build an explicit path to abstain. Low-resolution regions, clipped pages, glare, unreadable handwriting, contradictory totals: raise an exception. Confidence scores are model-specific and often miscalibrated; combine them with image-quality checks, format validators, cross-field arithmetic, and disagreement between recognition passes.

Provenance turns review from retyping into verification: store the page number and bounding polygon for every field that matters, and show the crop beside the proposed value. Without coordinates, that check means searching a whole page, or digging up the physical receipt.

What you let the model touch

A vision-language model earns its keep on harder reasoning: tying a label to a value elsewhere on the page, reconstructing a table split across a fold, explaining why two totals disagree. Keep observation and interpretation separate, and never let it write to a tool or database; validation and business rules stay outside the model.

Text printed on the document is untrusted input. A line that reads like an instruction is not permission to change extraction logic or call a tool nobody asked for.

Multi-page documents want the same discipline. Preserve page order and track which header belongs to which document, instead of handing the model a loose page bundle and trusting it to guess where one invoice ends and another begins. Split documents deterministically, attach identifiers up front, and state whether a table may continue across a page boundary.

Where this actually runs

Gate deployment on critical fields, not the aggregate score, and keep a stratified regression set of real failures: clean pages, common noise, difficult layouts, critical fields, a few impossible cases. Version the model, preprocessing, dictionaries, schema, and validation together, and replay the set whenever any changes.

Running this locally keeps documents under your control and gets cheaper as volume gets steadier, but you own storage, backup, model memory, accelerator utilization, and maintenance no spec sheet mentions. Hosted document APIs and frontier vision models buy elasticity and a higher exception ceiling, for upload time, retention, residency, and per-page or per-token cost. Hybrid routing, local for routine volume and hosted for flagged exceptions, works when privacy rules are strict and escalation stays visible.

Pick whichever deployment shape fits your situation; that part is negotiable. This part is not: never let a value reach a downstream system without the coordinates that prove where it came from.