Vision models for product-image analysis: prompts grounded in visible evidence

Point a vision model at a product photo and ask what's wrong with the packaging, and it answers confidently: a material guess, a damage story, an authenticity verdict. Nothing marks which words came from the pixels, and which from the model's sense of typical damage. That's the confidence problem. The model never sees your photo, only a resized, cropped, token-budgeted stand-in, and a fluent answer can be grounded, half grounded, or invented.

Don't call the model until the catalog rules fail

The job is narrower than "understand this image": describe attributes, compare listings, flag defects, retrieve similar items, from fixed views and catalog metadata. Embeddings and deterministic filters handle most of it alone. Call the vision-language model only for layout, ambiguity, cross-view relationships, or free text, and always ask the same thing: what's visible, where, how confident, then a conclusion.

One sentence, four different kinds of claim

Every clause carries different weight: "red mark in the corner" is observation, "that's a shipping label" is association, "the item was mishandled" is inference, "reject this unit" is action. Material, authenticity, exact dimensions, condition: pixels rarely settle any of them, so an open "what's wrong here?" rewards a story that matches expectation, not evidence. Require a region behind every conclusion, shown to the reviewer, not buried in a log. Watch for:

• an invented serial number
• a UI element outside the frame
• an identity claim across two different items

Score those apart from missing information: any can trigger a wrong action. Reward honest uncertainty. Label multi-image inputs as same object or sequence, drop duplicates, shuffle frame order: a story that survives isn't using the sequence.

What you log matters as much as what you ask

Log every transform before the model sees the image: resize, crop, tiling, the resolution setting the API used, not its default, even via Gemini's CLI multimodal mode. For a local model, say Ollama running a vision checkpoint, log the projector, quantization, and context length: a mismatched base model and projector can fail quietly while the server keeps talking.

model id, runtime, prompt version
image count, order, transforms
tokens in/out, latency, memory
verdict: grounded / unsupported / abstained

Score attribute precision, retrieval relevance, defect recall, and the unsupported-claim rate. Crop faces and addresses first, and retain originals and logs separately. A local model keeps images off someone's server, not an unencrypted disk. Time the whole path, hosted or local, not tokens per second. Stop tuning once the gate passes at acceptable cost: a bigger model isn't safer, just more detail to expose and more room to overreach.

What I haven't solved: verifying the citation itself. A model can learn to point at a region near a claim whether or not it's real, and a region that's always populated is decoration, not a check. I don't have a clean way to catch it yet.