OCR models for forms and handwriting: languages, scripts, and mixed alphabets

Store what the recognizer actually saw and what the business rule decided it means, in two separate fields, always. That's the whole rule. The rest of this is me talking you into it on paperwork where it's easiest to skip: applications, surveys, delivery slips, forms corrected by hand, printed labels sitting next to handwriting, check marks, names that won't survive contact with a spell-checker. A model good at reading a messy layout is also a model good at guessing, and a wrong guess on a surname costs a lot more than a wrong guess on a paragraph of prose.

The raw read and the clean read are different files

Start dumb, on purpose. Template registration, region cropping, a printed-text OCR pass, and a handwriting recognizer kept separate from it, because the two fail in nothing-alike ways. Bring in a generative or vision-language model only where it earns its keep: hard handwriting, an unfamiliar layout, a label three fields from its value, the exceptions the deterministic stages choke on. It doesn't get to replace stages that already work just because it's newer.

Everything that touches a page should leave a trail: file, page, capture device, the crop and dewarp and denoise settings, model and runtime version, language hints, and what a reviewer changed and why. None of that is optional once a page carries more than one language. Never let a cleanup step overwrite what was recognized: if a character could be an O or a zero, a decimal point, a minus sign, a unit, or a diacritic is ambiguous, keep the raw answer beside whatever the business rule guessed. Test with fixtures per language, mixed-script pages forced in on purpose, diacritics and names and numbers and punctuation scored separately, not folded into one error rate that hides which part broke.

A confident guess and a correct one look identical

The failure that actually bites is quiet: a legible handwritten value gets filed under the wrong field, or the model reads a mark that was never on the page. Neither looks broken. The trap on multilingual documents is running translation or spell correction inside the OCR step and losing the form the person actually wrote, because a language model continues a plausible pattern, and a damaged surname or part number gets completed into something that reads perfectly and is wrong.

Give the system somewhere to say it doesn't know. A clipped page, glare, unreadable handwriting, totals that don't add up: those should throw an exception, not a best guess with a confidence score attached, because those scores are model-specific and rarely calibrated the way you'd want. Pair them with image-quality checks, format validators, cross-field arithmetic, known-value dictionaries, and disagreement between two recognition passes. Store a bounding box for every field that matters, not just the value, so a reviewer sees the crop beside the proposed answer instead of hunting the page for it.

What the model is allowed to touch

A vision-language model can connect a label to a value that's nowhere near it, reconstruct a table that fell apart across a page break, or explain why two totals disagree, and that's useful. What it shouldn't do is write to a database without a schema and a business rule sitting outside it. Text printed inside a document is not an instruction, whatever it happens to say. Feed it pages one document at a time, identifiers attached, page order preserved; don't hand it a loose stack and trust it to notice where one invoice ends and the next begins.

Local deployment keeps documents inside your own walls and gets cheaper the more steady volume you push through it, once storage, backups, accelerator time, and review headcount are counted honestly. A hosted document API buys elasticity and a wider net for the genuinely strange page, at the cost of upload time, retention terms, data residency, and a per-page bill that adds up faster than budgeted. Hybrid routing works when the privacy line is firm and you can point to which documents cross it.

I still haven't found a clean answer for the case where the same person's name is spelled two legitimate ways across two scripts on two different documents. I punt it to a human queue every time, and I'm not convinced there's a better move.