Vision models for product-image analysis: choosing the right vision model

Pick the cheapest thing that can answer the question, and only promote to a vision-language model once it demonstrably can't. That's the rule. If you already run embeddings and catalog filters, prove with real cases that layout, ambiguity, or language pushed past what fixed rules handle.

A fluent answer from a vision-language model isn't proof of anything: it works from a resized, cropped image squeezed into a token budget, and a confident sentence can come from a detail that vanished before inference. Product-image work means catalog photos, uploads, packaging, defects, search queries: describe an attribute, compare two items, flag a bounded defect, retrieve something similar. Choice follows text density, spatial reasoning, image count, language, latency budget.

the cheap filter goes first

Build a blind evaluation set before opening a model card: real easy cases, ambiguous ones, a few impossible from the pixels alone, run at exact settings you'll ship with. Keep the original asset untouched and log every transformation before the call goes out. If a provider exposes a resolution flag, record the value sent; "default" is not debuggable in six weeks.

eval_record:
  model + runtime + prompt rev
  image: dims, crop, resize
  frames: count, order
  tokens: visual/input/output
  verdict: grounded | unsupported | abstain

For a local model, log the projector version, quantization, context window, and preprocessing library; weights from one release paired with a projector from another fail quietly, language staying grammatical while the grounding comes apart. gemini-cli-multimodal is a fast way to watch the request leave the machine, usually where a resolution assumption breaks; ollama-vision-local shows those choices mattering nearly as much as the weights.

Score attribute precision, retrieval relevance, defect recall, view consistency, then track unsupported claims separately from omissions. An invented serial number or a confident guess at authenticity can trigger a wrong action unnoticed. Put unanswerable examples in the set on purpose; score abstaining as a win.

from pixel to action, one shaky link at a time

The failure that shows up repeatedly is the model inferring material, authenticity, dimensions, or condition the pixels cannot establish, phrased the way a person would, so casual review misses it. The adjacent trap: reaching for a large general model when OCR, embeddings, or a small task-specific one already cleared the bar. Both mistakes survive a demo because the output matches what the reviewer expected.

Every conclusion should point at a region, page, frame, or labeled feature, visible to the reviewer, not buried in a log. Split reasoning into layers: "a red indicator at the upper right" is observation, "it belongs to the network panel" is association, "the connection has failed" is inference, "restart the gateway" is action. None should let a plausible observation jump to a consequential action. Put that rule into routing code, not prompt text, and version it for replay.

Multiple images add their own identity risk. Label each input, state whether they're the same object or an ordered sequence, and drop duplicate frames that eat token budget without evidence. Where order matters, attach timestamps and test a shuffled set: a model telling the same story after you scramble the frames isn't using the sequence, it's pattern-matching.

Then there's the machine underneath. Hosted APIs need upload time and retention policy tracked; local runs need cold-load time, projector memory, heat under concurrency. Compare the whole path at p50 and p95, not tokens-per-second alone. Crop what doesn't need to travel: faces, addresses, screens, other rooms. Set retention per artifact type; local only solves the transfer half of privacy, not an unencrypted disk or overbroad access.

Stop tuning once the gate passes at a latency and cost you can live with. A bigger model and more pixels aren't automatically safer; they mean more exposed detail and more room to answer past the evidence. I wouldn't bother fine-tuning a vision-language model: labeling pixels for a narrow catalog task costs more than one more embedding rule.

None of it holds if the humans downstream stop reading the abstained cases and start rubber-stamping whatever comes back confidently, which tends to happen when the pipeline's been quiet for a while.