OCR models for technical documents and labels: tables and key-value association

An engraved part number on a curved metal plate, shot under bad fluorescent light at an angle, reads correctly to a technician at a glance and reads as something else to a language model: something that looks more like a part number than the real one does.

That's the job. Manuals and clean invoices show up, but most of it is schematics, equipment plates, part labels, drawings, and maintenance records, and the target is a part number, a warning, a measurement, a callout, a symbol, and how they relate, not a paragraph. The page fights you: engraving instead of print, curved surfaces, tiny fonts, rotated callouts, line art standing in for text, near-identical identifiers.

A vision-language model can help, connecting a label to a value sitting elsewhere on the page or making sense of an odd layout. It can also quietly swap an uncertain character sequence for language that reads more naturally than the actual document, which is worse than garbage because nobody notices.

So the constraint I design around: preserve the association between a value and its evidence before that value maps into a business field. Start with high-resolution region OCR and dictionaries that check format without correcting identity. Bring in a generative or multimodal model only where it earns its keep: messy handwriting, an irregular layout, semantic field association, exception handling. Don't let it quietly replace the cheap deterministic stages that already work.

The log that has to exist before the label does

Treat the original file as immutable and assign a page and document identifier before anything touches it. Region OCR is also deciding what belongs to the page at all, not just what the characters say, and that decision needs to be recorded, not silently re-made on every reprocess.

• source file hash, page number, capture device
• original dimensions, orientation, color, plus crop, dewarp, threshold, or denoise version
• OCR, layout, or VLM model and runtime revision, plus language hints and dictionaries
• raw text, region coordinates, reading order, confidence
• normalized field, validation rule behind it, reviewer change, latency, memory, per-page cost

Skip that log and every disagreement six months from now becomes an afternoon of guessing. None of this is optional. The ambiguous-character problem lives here: O versus 0, I versus 1, a decimal point, a minus sign, a unit, a diacritic. When the recognizer isn't sure, store what it saw and what the business rule proposed as two separate fields. Normalization can propose a fix, but it doesn't get to overwrite what the recognizer actually saw.

Why word error rate lies to you about the account number

Character error rate and word error rate are fine numbers, but they weight a missing comma the same as a wrong account number, and that flattening is the failure this whole approach exists to catch. Add exact-match accuracy for critical identifiers, dates, and amounts. Track warning recall separately, since a missed safety warning is not the same category of miss as a missed adjective. Preserve unit correctness and region association as their own checks instead of folding them into one blended score, and track unsupported-normalization rate: how often a value changed without a validation rule licensing it.

For anything structured, score field association and table geometry on their own, separate from whether the transcription was correct. A table can transcribe every character right and still place the value under the wrong header, in the wrong row, and a character-level metric will never catch that.

A confident wrong answer is worse than a shrug

The failure that actually costs you is a model normalizing a part number, a decimal, a unit, or a safety label into a string that looks more familiar and is wrong. Language models are trained to continue plausible patterns, so a damaged part number or a smudged total gets completed into something close enough to the real answer to pass a glance. The specific trap is asking a model for final JSON in one shot: you get an answer, and you lose the evidence trail you'd need to find the swapped field later.

Build a path for the model to say it doesn't know. A low-resolution region, a clipped page, glare, unreadable handwriting, a total that doesn't add up: these should raise an exception, not produce a best guess dressed up as an answer. Confidence scores are model-specific and often badly calibrated, so pair them with image-quality checks, format validators, cross-field arithmetic, dictionary lookups, and disagreement between two independent recognition passes.

Provenance turns a review queue from retyping into verification. Store the page number and bounding polygon for every field that matters, then show the crop next to the value someone is checking. A person resolves an ambiguous digit in seconds when the region is right there. Take the coordinates away and the same review means searching a whole page, or reopening the physical document.

What the model reads on the page isn't a request

A capable VLM can link a label to a value sitting far away, reconstruct a table split across a page break, or explain why two totals disagree. That's useful. Ask it to keep raw observation separate from interpretation, and keep any tool or database write behind schema validation and business rules that live outside the model entirely. Text printed inside a document is data someone else wrote. It doesn't get to change how the extraction runs or trigger a tool call just because it asked.

Multi-page handling deserves the same discipline. Preserve page order and repeated-header identity, and don't hand the model an arbitrary bundle of pages and trust it not to merge two invoices into one. Split documents deterministically, attach identifiers, and state whether a table or paragraph can continue across a page boundary. Extract structure and provenance before business normalization, and version the model, preprocessing, dictionaries, schema, and validation together. Change any one of them and replay a stratified test set: clean pages, common noise, the hard layouts, the fields that actually matter, and a few that should be unreadable.

Where you run this matters too. Local deployment keeps the documents under your own control and gets cheap once volume is high and steady, but you're carrying storage, backup, model memory, accelerator time, maintenance, and review yourself. A hosted document API or a frontier vision model gives you elasticity and a higher ceiling for the weird exception, at the cost of upload time, retention settings, data residency, rate limits, and a per-page or per-token bill that never stops. Hybrid routing earns its complexity when privacy rules are strict and escalation stays visible.

I run local by default for this kind of work, because I'd rather own the documents and the cost curve than rent someone else's ceiling. What that trade costs me is real: less elastic burst capacity when volume spikes, and a lower ceiling on the strange exception than a frontier hosted model would give me. I made that trade on purpose, and I'd make it again, but I'm not going to pretend it comes free.