OCR models for invoices and receipts: preprocessing for OCR models

A reconciliation run flags an invoice as paid twice: the source image shows a smudged decimal point rewritten into a comma borrowed from the subtotal three lines up. Nobody typed the wrong number. A model did, confidently enough that nobody checked. Any team running receipts through OCR at volume has some version of this, usually caught by an auditor, not the pipeline.

Treating OCR as solved, point a model at an image, get text back, is too small a frame here. The system decides what belongs on the page, what order it reads in, whether the characters are trustworthy, and how each value maps back to the source. Inputs arrive messy: crumpled invoices, faded thermal receipts, expense slips shot at an angle, shadowed. The output is a schema: vendor, dates, totals, tax, currency, line items, coordinates back to source.

A vision-language model earns its place linking a label to a value far off on the page, rebuilding a shredded table, explaining why two totals disagree. It can also do worse: swap an ambiguous character run for language that reads more naturally than what was printed. Deskewing and contrast fix legibility problems; push either hard enough and you manufacture characters that were never there, or erase faint ones that were. The rule that survives production: easier to read, not different in what it says.

the original stays in the folder

Start with plain OCR, then vendor templates, anchors, arithmetic checks. Bring a generative or multimodal model in only where it earns its keep: handwriting, irregular layouts, semantic field association, the exception pile deterministic stages can't clear. Don't retire a working stage for being newer.

Keep the source file immutable, stamp every page with an identifier first. Version each preprocessing step, keep it reversible, compare against the untouched original. When any piece changes, replay it against a fixed set: clean pages, common noise, hard layouts, critical fields, and one built to fail. Where a character is ambiguous, an O that might be a zero, a minus sign lost in a fold, store what the recognizer saw and what the business rule guessed as separate values, never one overwriting the other.

Worth logging:

Character and word error rate track legibility, but they'll call a pipeline healthy while it swaps an account number for a different, equally well-formed one. Add exact-match checks on the fields that matter, identifiers, dates, amounts, and score table geometry and field association apart from raw transcription. The gate that matters: field accuracy, arithmetic reconciliation, duplicate detection, exception rate.

confident is not the same as correct

Garbled output is the easy failure to catch. The dangerous one is a syntactically perfect number lifted from the wrong line: subtotal instead of total, balance-forward instead of amount due. Nothing about it looks broken. Language models continue plausible patterns, so a damaged surname or a smudged part number gets completed into something familiar-looking instead of flagged. Push a page too hard toward visual neatness and you lose decimal points, faint handwriting, and table-cell boxes, while the result looks cleaner than it ever was.

Build an abstention path and mean it: low-resolution regions, clipped pages, glare, unreadable handwriting, totals that won't reconcile should raise an exception, not a guess. Confidence scores vary by vendor and are poorly calibrated, so pair them with image-quality checks, format validators, arithmetic cross-checks, disagreement between passes. Store the page number and bounding polygon for every important field and show the crop next to the proposed value: verification, not retyping. Skip the coordinates and a reviewer hunts a whole page, or the physical receipt.

When a vision-language model reasons over a document, keep observation and interpretation apart, and keep tool or database writes behind schema validation outside the model. Text on the page is untrusted data, not permission to change the extraction instructions or trigger a tool call. On multi-page batches, split deterministically, attach identifiers, preserve page order and header identity, and state whether a table continues across a page boundary, rather than letting the model sort a loose stack itself.

Run it locally and documents stay under your control, cheaper as volume climbs, once you count storage, backup, accelerator time, and review headcount. A hosted API buys elasticity and a higher ceiling for weird exceptions, at the cost of upload time, retention terms, residency, and per-page pricing that adds up fast. Hybrid routing, local first with hosted escalation, works when privacy rules are strict and escalation stays visible in the audit trail. None of that replaces judgment: every rule here still won't save you from the one document type nobody thought to test.