OCR models for invoices and receipts: handwriting mixed with printed text

Feed every page of an invoice batch through a vision-language model and the accelerator, or the API meter, becomes the bottleneck before accuracy does.

Use it only where it earns its keep: hard handwriting, odd layouts, a field that needs the right label, an exception nobody wrote a rule for. Print and handwriting are different problems; one threshold rarely fits both. Start with OCR, templates, anchors, arithmetic checks. Cheap, and it works.

Freeze the evidence before you touch it

Segment by type before interpreting anything: print, handwriting, marks, blanks. Decide what a region is before what it says, then freeze the file with a page and document id:

file hash + page id + capture device
dimensions, color, orientation
crop / dewarp / threshold / denoise, versioned
OCR / layout / VLM model + runtime revision
raw text, region, reading order, confidence
normalized value, validation result, reviewer edit
latency, memory, energy or per-page spend

If a character could be an O or a zero, a decimal or a stray mark, store what the recognizer saw next to the rule's output, and never overwrite one with the other. Character error rate is a poor gate here; it treats a missing comma like a wrong account number. Score exact matches on critical fields, and grade tables separately.

The failures that don't look like failures

The dangerous OCR mistake is never the string that's obviously wrong. It's the syntactically perfect amount lifted from the subtotal or a balance-forward line, because a language model continues a plausible pattern and a smudged total looks like a clean one it has seen before. A strong printed score can mask a bad handwriting read, or misassign a note to the nearest label.

Give it a way to abstain, and route these to a person:

• resolution too low to trust a digit
• a clipped page or missing corner
• glare or shadow across a field that matters
• handwriting confidence indistinguishable from print
• two totals on one page that disagree

Confidence scores alone are poorly calibrated: pair them with image-quality checks, format validators, arithmetic, and disagreement between passes. Store a bounding box per field and show the reviewer the crop beside the value: review becomes verification, not retyping.

What the model gets to touch, and what it doesn't

A vision-language model earns its cost when it links a label to a distant value, rebuilds a table split by a fold, or explains a mismatch between two totals. Let it observe and propose, not decide. Database writes need schema validation and business rules outside the model: text on a document is untrusted input, not an instruction.

Across a multi-page batch, keep order and header identity explicit, split documents deterministically, and state if a table crosses a page break. Version the model, preprocessing, dictionaries, and validation together, and replay a fixed set of clean, noisy, and known-impossible pages when one moves.

Local processing keeps documents under your own roof and gets cheaper at volume, once you count storage, accelerator time, and review. Hosted APIs trade that control for elasticity and a higher exception ceiling, for upload time, retention, per-page cost. Route by sensitivity, keep escalation visible.

What I'd check next: pull the exception queue and see how many entries are unreadable versus how many are the model guessing instead of flagging. If the ratio skews toward guessing, fix the abstention threshold, not the model.