OCR models for tables and statements: handwriting mixed with printed text

A statement page arrives with a printed header, typed line items, and a handwritten correction scribbled in the margin. One image, three kinds of evidence, and a pipeline that has to call all three before it knows what that margin note means.

That's the actual job. Not image-to-text, mostly solved for a clean printed page, but deciding what belongs on the page, what order to read it in, which characters you can trust, and how each value traces back to a pixel someone can point to later. Tables and statements make this harder than a receipt: financial statements, schedules, price lists, lab results, multi-page tabular reports where a wrapped cell, a repeated header, an indentation level, or a negative-number convention all carry meaning plain text throws away, and faint rules and dense small type don't help.

A vision-language model can untangle a lot of that. It can also lie to you politely, replacing an uncertain character sequence with something that reads more naturally than the document says, without hesitating.

Print and handwriting don't earn the same trust

Treat print, handwriting, marks, and blank fields as separate problems from the start, because the pipeline needs to know which one it's looking at before it decides how much to trust the result. The base layer should be boring on purpose: layout-aware OCR with table detection, checked against deterministic spreadsheet validation, covering clean printed cells cheaply and predictably. A generative or multimodal model earns a place only where it beats that baseline on something measurable: difficult handwriting, an irregular layout, tying a label to a value three columns away, an exception the deterministic pass can't resolve. It should never quietly replace a stage that already works.

The practical test: segment regions by type, evaluate the writer or instrument behind each one, route low-confidence handwriting to a person before it reaches a database. Keep the source file immutable and assign a page and document identifier first: once cropping and dewarping start you lose the way back to the original.

Then log the whole path from capture to output, not just the final values:

• source hash, page number, capture device
• crop, dewarp, threshold, and denoise versions applied
• OCR, layout, and VLM model and runtime revision
• raw text, region, reading order, confidence, normalized value, reviewer change

Normalization should never overwrite transcription. If a character could be an O or a zero, a capital I or a one, a decimal point or a stray mark, store what the recognizer actually saw and what your business rule proposed as two separate values. Collapse them on first touch and a decimal error survives review after review, unnoticed.

Judge the output on cell text, row-column association, header hierarchy, numerical consistency, and how well the table reconstructs, not on character error rate alone. CER and word error rate will score a missing comma the same as a wrong account number, which is nonsense on a statement. Add exact-match scoring for identifiers, dates, amounts, warnings, and units, and score field association and table geometry as their own metric.

The wrong row is worse than the wrong digit

The failure that actually costs money isn't garbled text. It's correctly recognized digits attached to the wrong row, column, period, or unit, and nothing about it looks broken. A model with a strong printed-text score will let that confidence bleed onto the handwritten field next to it, or assign a stray note to whichever label sits closest, because that's what these models are trained to do: continue the most plausible pattern. A damaged surname, a part number, a total, all get completed into something that looks right and isn't.

Build an actual path for the pipeline to give up. Low-resolution regions, clipped pages, glare, unreadable handwriting, totals that don't add up, should raise an exception, not a best guess dressed up as an answer. Confidence scores from the model itself are model-specific and often poorly calibrated, so combine them with image-quality checks, format validators, cross-field arithmetic, dictionary lookups, and disagreement between two independent recognition passes.

None of that matters if a reviewer can't act on it fast. Store the page number and bounding polygon for every important field, and show the crop next to the value. A reviewer resolving one uncertain digit against a visible crop takes seconds. Hunting for that field across a whole scanned page, or pulling out the physical document, takes considerably longer, and eventually the reviewer just approves whatever's on screen.

When you let a vision-language model reason over a page, keep raw observation and interpretation in separate lanes. Any write to a database or downstream tool needs schema validation and business rules outside the model. Text printed inside the document is untrusted input, not an instruction to act on, even when it looks like one.

Multi-page work raises the same question at a different scale. Preserve page order and repeated-header identity, and don't hand the model an arbitrary bundle of pages and hope it keeps invoices and statements from bleeding together. Split documents deterministically, attach identifiers at the split, and state whether a table or paragraph may continue across a page boundary. Leave that unwritten and the model makes its own call, quietly.

Score handwriting independently from print, keep a genuine unreadable state instead of forcing a guess, and version the model, preprocessing, dictionaries, schema, and validation rules as one unit. When any of those changes, replay a stratified test set: clean pages, common noise, awkward layouts, the fields you care about, and a few that should fail outright.

Local deployment keeps documents under your control and gets economical once volume is high and steady, but you pay in storage, backups, model memory, accelerator time, and maintenance nobody budgets up front. A hosted document API or frontier vision model buys elasticity and a wider exception ceiling, at the cost of upload time, retention settings, residency, rate limits, and a per-page or per-token bill. Hybrid routing, local for the easy majority, hosted for flagged exceptions, holds up fine once privacy rules are firm and escalation is visible.

Before trusting any of this on a new document type, pull a stack of pages that mix handwriting into printed tables and check one thing: does every flagged field arrive with a crop next to it, or does the reviewer have to go hunting for it. If it's the second one, the review step doesn't work yet, whatever the accuracy number on the dashboard claims.