Vision models for chart and diagram understanding: an evaluation set for visual reasoning

You crop a dashboard panel down to one line chart, paste it into a vision model, and ask whether the trend is still climbing at the right edge. The model answers right away and sounds sure of itself. It never saw your image, only a resized, tiled, token-budgeted stand-in for it, and nothing in the response says which one it read from.

The material here is charts, plots, dashboards, architecture diagrams, schematics, slide visuals: meaning lives in position, color, and small print, not in flowing prose. The job is to recover the relationship on the page and answer one specific question without guessing at a label the model can't read. Feed the eval the original image or a vector export, the caption or legend beside it, the units, and the exact question a person would ask. That's narrower than 'understand this image', and narrower is what makes it measurable.

Before any of that, check whether you need a vision model at all. If the underlying chart data or the graph's source structure is sitting right there, use it: a non-generative baseline beats a VLM on cost and reliability every time. Bring in visual reasoning only once the baseline hits a wall: layout ambiguity, a relationship the data format doesn't encode, cross-referencing a legend against a point on the page, or language buried in the pixels.

The image the model actually gets

Label which region of the image actually carries the answer, then build controlled perturbations around it: blur, glare, an occluded corner, a crop that clips the legend, text shrunk past what anyone could read. Blind the outputs before scoring, because knowing the answer changes what you're willing to accept. Preserve the original asset and log every transformation applied before the model saw it. Score unsupported claims separately from missing information: different failure classes, conflating them hides which one you have. The record needs more than a model name attached to a pass or fail.

model ID, API/runtime, prompt revision
original dimensions, crop, resize, orientation
image count, ordering, frame timestamps
OCR or metadata supplied beside the pixels
visual/input/output token accounting
first response time, total latency, peak memory
grounded pass, unsupported claim, abstain, repair

If the provider exposes an image-detail or resolution setting, log the value it used, not the default you assumed. Scripting the hosted side, testing that flag through a multimodal Gemini CLI setup beats trusting whatever default the SDK chose. Running locally, log the vision projector, quantization, context length, and whatever preprocessing resizes the image before the encoder sees it. Text weights from one release paired with a projector from another can degrade quietly while the server keeps returning fluent language: a broken pairing never announces itself.

The acceptance gate is label transcription, relationship accuracy, numerical consistency, evidence citation, and abstention on anything unreadable. An omission earns a second look. An invented serial number, a UI control that isn't there, a trend that isn't in the data: that kind of claim can drive a wrong action unnoticed. Put a handful of unanswerable examples in the set and reward the model for saying so.

Confident answers that never touched the pixels

The classic failure is a confident, well-formed description of a trend that has reversed an axis, a series, a unit, or an arrow direction. It reads as correct because it matches what a person expects to see, and that's what lets it slip past review. A related trap: benchmarking on public images that look nothing like your production captures, low-res crops, phone glare, an angled whiteboard shot. Both survive review the same way. Fluency reads as grounding. Require a region, a page, a frame, or a labeled feature behind every conclusion that matters, and show that provenance to the reviewer instead of burying it in a log file.

Split the prompt, and the scoring, into four layers, because each one needs different evidence and a different amount of trust:

• observation: a dot sits in the corner of the panel
• association: that panel belongs to the network health section
• inference: the link is down
• action: restart the switch

Don't let a plausible observation walk straight into an action step. That chain from dot to restart command should require a human, or at least a second check, not one confident paragraph.

Multiple images add an identity problem on top of the grounding one. Label each input explicitly: same object, a page sequence, one camera, one time period, since the model will happily assume continuity you never asserted. Drop duplicate frames: they burn tokens, add no evidence. Where order matters, attach timestamps and run a reordered control. If shuffling the frames doesn't change the story, the model wasn't using the sequence.

None of this should stay as prompt wording for long. The rule deciding when a vision model is trustworthy enough to promote, past real capture conditions and calibrated abstention, belongs in preprocessing, routing, or authorization code, not a prompt someone can edit by accident. Store the model and pipeline version with every result so an upgrade can be replayed against the same images.

Privacy sits on both sides of the call. Crop rooms, faces, screens, addresses, and reflections before the image goes anywhere, and set separate retention windows for originals, thumbnails, OCR text, embeddings, prompts, and debug captures: they don't all need the same retention. Running the model locally, through something like an Ollama vision setup, keeps the bytes off someone else's server, but it does nothing for an unencrypted disk or an overbroad dashboard account.

For hosted calls, test upload time, regional routing, retention settings, rate limits, and provider errors alongside the answers. For local runs, measure cold load time, projector memory, CPU/GPU placement, preprocessing time, heat, and concurrency, not a single warm request. Compare the whole path at p50 and p95, not the tokens-per-second figure on the model card.

Stop tuning once the gate passes at a latency and cost you can live with, and the abstention rate looks calibrated rather than lucky. A bigger model or a higher resolution setting isn't automatically safer. It can surface more private detail per request and make the system more willing to answer past what the evidence supports.

What I'd check next: pull a stack of transcripts, cover the answer, and see whether you can point to the exact pixels that earned it. If you can't, the model couldn't either.