Vision models for product-image analysis: an evaluation set for visual reasoning
A vision-language model never looks at the photo you uploaded. It looks at whatever survived the runtime's resize-and-tile step, plus whatever token budget went to pixels instead of text. Change the crop or run short on context, and the model answers questions about a different image than yours, and nobody catches it in review because their copy looks fine.
For product-image work the material is catalog photos, phone uploads, packaging shots, and defect photos, and the job is narrow. Describe an attribute. Compare two listings. Flag a bounded defect. Retrieve something visually similar. Feed the model controlled views, catalog metadata, allowed attribute values, and a quality signal on the photo, and a fuzzy visual question turns into something you can score.
Try the boring filter first
Run the non-generative baseline first: image embeddings plus deterministic catalog filters. It's dull. It's cheap. For SKU match, color family, or duplicate detection, that baseline is also correct. A generative model earns its slot once the baseline hits layout, ambiguity, relationships between objects, or language fixed rules can't cover cheaply. Skip straight to the VLM, and you'll spend the evaluation budget re-discovering that embeddings already solved half your test set.
What belongs in a test record
Build the set around real capture conditions, not polished catalog shots: blur, glare, occlusion, tiny text, bad crops, plus examples with no correct answer. Label the region that carries the answer, apply controlled perturbations to it, and blind the outputs before anyone scores them. Preserve the original asset and log every transform applied before the call goes out. "The model was wrong" and "the model never saw the label" are different bugs.
A model name and a pass/fail flag aren't a record:
| Field | Why you keep it |
|---|---|
| Model ID, runtime, prompt revision | reruns need the exact config |
| Original dimensions, crop, resize, orientation | most common silent cause of a bad answer |
| Image count, ordering, timestamps | tests whether sequence matters |
| OCR or metadata beside the pixels | text versus pixels |
| Token accounting: visual, input, output | shows truncation |
| Latency (first response, total), peak memory | matters more than tokens/sec |
| Grounded pass, unsupported claim, abstain, repair | the real outcome categories |
If the provider exposes an image-detail or resolution setting, log the value you sent, not the default you assumed; scripting calls through gemini-cli-multimodal makes that value visible. Locally, record the projector, quantization, context window, and preprocessing; running that stack through ollama-vision-local keeps settings pinned. Mismatched text weights and projector versions degrade quietly while the server keeps answering in fluent sentences.
Four rungs from pixel to action
The failure that survives review is the model inferring material, authenticity, dimensions, or condition the pixels can't establish, dressed in a sentence that matches what a person expects to see. Testing on public benchmark images makes this worse: their composition looks nothing like production photos, and a model tuned to the benchmark's style scores well on exactly the wrong evidence.
Split every answer into four rungs and demand different evidence for each: observation ("a red mark, upper right"), association ("that's the low-battery icon"), inference ("battery's likely dead"), and action ("replace it"). Don't let a plausible observation jump straight to the action rung. Score unsupported claims separately from omissions: an omission gets flagged for review, an invented serial number or defect can trigger a wrong action. Put unanswerable examples in the set, and reward the model for saying the evidence isn't there.
Multiple photos, one identity problem
Multi-image inputs add a failure mode single images don't have: identity confusion. State whether the frames show the same object, a page sequence, or different cameras and times, and strip duplicate frames that burn tokens for no evidence. When order matters, attach timestamps and run a reordering test: shuffle the frames and rerun the prompt. The same causal story either way means the model isn't reading the sequence, it's pattern-matching frames and improvising the rest.
Running it without leaking the warehouse
Crop out what doesn't need to travel: faces, screens, addresses, reflections, rooms that aren't the product. Set retention separately for originals, thumbnails, OCR text, embeddings, and debug captures; none of them need the same lifespan. Running it locally cuts external transfer, but does nothing for an unencrypted disk or a dashboard account with too much reach.
For a hosted API, put upload time, regional routing, retention settings, and rate limits into the test too. For a local model, add cold-load time, projector memory, CPU or GPU placement, and concurrency, and compare the whole path at p50 and p95, not tokens-per-second. Stop tuning once the accuracy gate, the latency budget, and the abstention rate clear together. A bigger model with a bigger image budget isn't automatically safer; it can expose more of the photo and get more confident about details the pixels never supported. I wouldn't bother swapping in a heavier model because a leaderboard says it scores higher on some unrelated task.
Next thing I'd check: pull a handful of your multi-image product sequences, shuffle the frame order, and rerun them against whatever model you're about to ship. If the causal story stays the same, that tells you whether the sequence is being read at all, or just decorated after the fact.