OCR models for forms and handwriting: preprocessing for OCR models
A stack of intake forms lands on the scanner: some typed, some in shaky ballpoint, one crossed out and rewritten. An unmarked checkbox, a total that doesn't add up. Not a text-recognition problem. A decision problem: what belongs on the page, what order to read it, whether the output can be trusted.
My rule: cleanup should make the page easier to read without changing what it says. Deskew, denoise, contrast, threshold: fine, that's photography. Guess a missing stroke and you've started inventing.
keeping the boring stages boring
Start with template registration, region cropping, printed OCR, and a handwriting recognizer running on its own: cheap and predictable. Bring in a vision-language model only where it earns its keep, hard handwriting, odd layouts, a label far from its value, never to replace a stage that already works.
Version every transform against the untouched original, not the previous pass, and stamp a document ID first. Keep, per field:
- what the recognizer saw and its confidence, separate from the normalized value
- which transform and model produced it
An ambiguous O/zero or decimal/minus stays two values, not one merged guess.
the value that's correct for the wrong field
The failure that costs you isn't garbled text. It's a well-formed value attached to the wrong field, or a mark read into existence on a blank page. Models continue plausible patterns: a smudged surname or damaged part number gets completed into something plausible, and wrong.
Build an abstention path: low resolution, glare, unreadable handwriting, or a total that won't reconcile should raise an exception, not a guess. Confidence scores alone aren't enough: pair with image-quality checks, format validators, cross-field arithmetic.
Store the bounding box per critical field and show the reviewer the crop beside the value: verification, not retyping. Score exact matches on dates, amounts, and identifiers, since error rate treats a comma and an account number the same.
what the model gets to touch
A vision model earns its place linking a label to a distant value. Ask it to separate observation from inference. Its output shouldn't touch a database without schema validation outside the model, since text on a page is untrusted, not an instruction.
Preserve page order across multi-page batches, don't merge two invoices just because they're consecutive, and say whether a table may continue across a boundary.
Local deployment keeps documents under your control and gets cheaper as volume climbs, storage and review counted in. A hosted API or frontier vision model buys elasticity and a higher exception ceiling, at the cost of upload time and a per-page bill.
None of this is free. An audit trail this thorough is itself work to maintain, and on a form touched twice a year it's more infrastructure than the problem deserves. Sometimes the cheapest option is pointing a general model at the page and reading the output with a skeptical eye.