Vision models for product-image analysis: structured output from images
Feed a vision-language model a shelf photo and ask for a structured attribute record, and the JSON parses clean nearly every time. That's not the win it looks like. A valid schema proves the model follows instructions, not that "burgundy" or "cracked corner" came from the pixels rather than a guess. The model sees a compressed stand-in for your photo, and a fluent answer can be grounded, half grounded, or built around a vanished detail.
Narrow the job first. For product images the work is describing attributes, comparing listings, flagging a bounded defect, or pulling similar items from a catalog, not "understanding a photo" in general. Feed it controlled views, catalog metadata, allowed values, and a quality signal, and every call becomes measurable.
Earn the right to call a model at all
Run the boring baseline first: image embeddings plus deterministic catalog filters, cheap and enough for most retrieval and comparison work alone. Generative reasoning earns its place only when the baseline hits layout, ambiguity, or language fixed rules can't handle economically. Once a VLM is justified, keep the original asset, validate structure in code rather than prompt prose, and log every transformation before the call. The record worth keeping: runtime and prompt revision, crop, resize, orientation, frame timestamps, token accounting, and a verdict of grounded, unsupported, abstain, or repair. Log the image-detail control's actual value, as Gemini CLI's multimodal mode exposes it, not the default. A local vision model adds its own list: projector, quantization, context length, preprocessing code, because mismatched weights and projector fail quietly while the server keeps talking.
The answer that survives review anyway
A schema-valid answer and a true answer are not the same claim.
The failure that matters: a model inferring material, authenticity, or condition the pixels cannot establish, then handing you a hallucination that walks into a database looking valid. It survives review because it matches what a person expected to see. Demand a visible feature behind every consequential conclusion, and show it to the reviewer instead of a log. Separate observation, association, inference, and action in the prompt: a red indicator is observation, that it belongs to the network panel is association, that the connection failed is inference, restarting the gateway is the action, and each layer needs separate evidence before it can trigger something irreversible. Multiple images add identity risk too: label each one, say whether they're the same object or a sequence, drop duplicate frames that add nothing, and test a shuffled order, because a model that repeats itself after shuffling isn't using the sequence.
Score unsupported claims apart from missing information: an omission earns review, an invented attribute can trigger a wrong action alone. Build unanswerable examples into the test set and reward the model for admitting it. The acceptance gate is attribute precision, retrieval relevance, defect recall, and consistency across views. Crop what doesn't need to travel, rooms, faces, addresses, reflections, and set retention separately for originals, thumbnails, and embeddings: a local model keeps data off the wire but does nothing for an unencrypted disk. Compare the whole path at p50 and p95, not tokens per second: hosted owes upload time and rate limits, local owes cold-load and heat. Stop tuning once the gate clears.
I'd rather ship a system that abstains too often than one that answers everything. That costs throughput and a bigger review queue, and I'll take that over a customer finding out the "genuine leather" tag was a guess.