Vision models for chart and diagram understanding: video through frame sampling

And that's the whole problem: the model never looked at your chart. It looked at whatever survived the resize, the crop, the tiling, and the token budget, and answered with equal confidence. You can't tell what it saw from what it invented.

Point one at a dashboard, a plot, an architecture diagram, and the job is narrow: recover the relationships and answer one question without guessing at an unreadable label. Feed it the original image or a vector export, the caption, legend, units, and exact question, not "understand this image," which invites an answer beside the point.

Check first whether you need a vision model. If the chart has underlying data, or the diagram a source graph, use that: a parser reads axis values correctly, a VLM doesn't. Bring the model in once ambiguity outruns what a parser can express. I've spot-checked this from a multimodal CLI, never a real eval. With video, the decision is which frames it sees, not which model: coverage and event boundaries beat a stack of near-identical frames.

the record you need before you trust an answer

Write down more than the model name: dimensions, crop, image count and order, frame timestamps, and token accounting split into visual, input, output. Note first-response and total latency, and classify the answer as grounded pass, unsupported claim, abstain, or repair. A local VLM adds one line: the vision projector and quantization. Mismatched weights and projector answer fluently while seeing nothing, silently. Skip the full record for a one-off; keep it for anything unattended.

reward the abstain

The gate checks label transcription, relationship accuracy, numerical consistency, evidence citation, and abstention on the unreadable. Score unsupported claims apart from missing information; the two fail in opposite directions. Saying nothing sends a human to look, annoying but safe. Inventing a serial number or a trend becomes fact downstream. Reward the model for admitting it can't tell.

the axis flips and nobody notices

The failure that costs you is a confident trend that quietly reversed an axis, a series, or a unit. It reads like a correct answer and survives review because it matches expectation. Video has its own version: sample at a fixed slow interval and the model takes the blame for missing an event between frames, when the bug sat in your sampler. Require a region or frame per conclusion, and show that citation, not a log.

Separate what the model does into layers. "Red indicator, upper right" is observation. "Belongs to the network panel" is association. "The connection has failed" is inference. "Restart the gateway" is action. Each wants different evidence and authority. Don't let a plausible observation license an irreversible action.

frames are not free

Multiple images bring their own risk: identity. State whether frames share the same object, sequence, camera, or time period, or the model will connect images that were never related. Strip duplicates; they cost tokens for nothing. Timestamp frames when order matters, and test a shuffled control: a model telling the same story after you scramble it isn't using the sequence, it's pattern-matching the set.

Which is the sampler problem again. Design it around event duration, not a clock tick, and reach for a video-native model only once temporal reasoning earns its cost over frame-by-frame. Put that rule into code, not prompt prose, and store the model and pipeline version with every result.

privacy and the p95 nobody benchmarks

Privacy is a before-and-after problem. Crop the room, the face, the screen, the address, wherever you can, before the image goes near a model. Set retention separately for originals, thumbnails, OCR text, embeddings, prompts; each leaks on its own timeline. A local model keeps bytes off someone else's network but does nothing for an unencrypted disk.

For a hosted API, test upload time, regional routing, and rate limits alongside accuracy. Local is closer to serving vision models through Ollama: cold load, projector memory, GPU placement, heat, concurrency. Measure the whole path at p50 and p95, not tokens per second.

Stop tuning once the gate passes at a latency and cost you can live with. A bigger model and more pixels aren't automatically safer: more private detail exposed, a system readier to answer past the evidence.

If I kept exactly one rule out of all this: build the sampler around when things happen, not around a clock, and don't reach for a video-native model until frame-by-frame has let you down.