Vision models for chart and diagram understanding: choosing the right vision model
...so no, that fluent answer about your dashboard screenshot is not proof the model actually read it.
It never saw the image.
It saw whatever survived resizing, cropping, tiling, and the token budget the runtime spent that day. A confident answer can be fully grounded, half grounded, or built on a label cropped off before inference, and those look identical from outside. A leaderboard score won't tell them apart: it was scored on someone else's crop and resolution.
Narrow the job before picking a model. For dashboards, plots, diagrams, and slide visuals, you need relationships recovered and a question answered without guessing at an unreadable axis: the image, caption, legend, units, and the exact question, not a vague prompt. Check first whether you need a vision model: if the chart's data or source graph exists, use that. A VLM earns its place once layout, ambiguity, relationships, or language rule out the cheaper path.
Log the crop before you trust the caption
Model choice turns on text density, spatial reasoning, image count, language, latency, and deployment constraints. Build a small blind set: easy, ambiguous, impossible cases, tested at the exact settings you ship, an API path like Gemini CLI's multimodal mode, or a local runtime like Ollama's vision models. Log every transformation: dimensions, crop, resize, ordering, OCR or metadata, token accounting, latency, memory, and the resolution control's value. Local setups need the projector, quantization, and context length logged: mismatched weights and projector fail quietly, the server still answers. Score transcription, relationship accuracy, numeric consistency, cited evidence, and abstention, and treat an invented serial number as worse than an honest can't-tell.
Axes reverse, frames shuffle, and the answer still sounds right
The classic failure is a convincing trend sitting on a reversed axis, swapped series, or flipped arrow, and it survives review because it matches what reviewers expect. The quieter trap is reaching for the flagship model when OCR, embeddings, or a small task-specific VLM clears the bar for less money and exposure. Make every conclusion cite a region, page, frame, or feature, and show that provenance instead of burying it in a log. Keep observation, association, inference, and action separate: a red indicator is not restart the gateway, and jumping straight to it restarts the wrong thing. With multiple images, label each one, state whether it's the same object or a sequence, drop duplicates, and shuffle order as a control: a model telling the same story after reordering isn't using it.
Put that rule, smallest model that clears the hardest case and abstains on the rest, into routing and validation code, not prompt text. Crop faces and screens before storage, not just display, and set retention separately for originals and thumbnails; a local model keeps pixels off someone's server, it doesn't encrypt your laptop. Measure the whole path, upload, preprocessing, generation, at p50 and p95, not tokens per second. Stop once the gate passes at a cost you can accept. I'd rather trust a smaller model that admits it can't read something than a bigger one that never does, though I've shipped the bigger one when the deadline mattered more than the principle.