Vision models for product-image analysis: video through frame sampling

Here's the rule, before I spend the rest of this earning it: build your frame sampler around event boundaries and scene changes, never a fixed clock interval, and don't call a video-capable vision model until embeddings and deterministic catalog filters have already failed you on the same question.

Product-image work is narrower than "understand this image," and that narrowness is the trick: observable attributes, item comparison, a bounded defect, visually similar products from controlled views plus metadata. Keep the question tight and you can measure whether the model answered it.

A vision-language model never sees your image. It sees a resized, cropped, tiled, token-budgeted stand-in, and the crop happened before the model got a vote. A fluent answer can be fully grounded, partly grounded, or built around a detail discarded on the way in, and from outside those three look identical.

Every frame needs a receipt before it reaches the model

Sample video around motion and scene-change signals, keep before-and-after context around each boundary, and check event recall against clips you've annotated. Then keep the receipt: preserve the original asset and log every transform applied on top of it. "The model was wrong" and "the model never received the evidence" are different bugs, and you can't tell them apart without a record.

model id, api or runtime, prompt revision
original dimensions, crop, resize, orientation
frame count, ordering, timestamps
OCR or metadata attached alongside pixels
input and output token accounting
latency to first token, total latency, peak memory
grounded / unsupported claim / abstain / repair

If the provider exposes an image-detail or resolution knob, log the value you got, not the default you assumed. Running local, log the vision projector, quantization, and context window too: mismatched weights and projector can degrade quietly while the server keeps producing readable sentences. Local-runtime notes are in wiring a vision model into Ollama; the hosted side, including where that detail parameter lives, is in driving multimodal input through the Gemini CLI.

Grade against attribute precision, retrieval relevance, defect recall, cross-view consistency, and unsupported-claim rate, scored on its own axis. A missing answer sends someone back to look again. An invented serial number or fabricated dimension gets acted on. Reward "I can't tell from this image" as correct, not as a failure to respond.

The gap between what you see and what you conclude

The failure that costs you is the model inferring material, authenticity, dimensions, or condition the pixels never established, said in the same confident tone as everything else. It survives casual review because it matches what a person expected to see. The video-specific version: sample at a slow fixed interval, then blame the model for missing an event that happened between two frames it was never shown.

Split conclusions into four layers and don't let one borrow another's authority: "there's a red indicator, upper right" is observation; "it's on the network panel" is association; "the connection has failed" is inference; "restart the gateway" is action. An observation should never jump straight to an irreversible call.

Multiple images multiply this risk. State whether inputs show the same object, a page sequence, or different cameras, and strip duplicates that burn tokens without adding evidence. If order matters, attach timestamps and run a shuffled control: a model that tells the same story after you reorder the frames isn't using the sequence, it's pattern-matching on the set.

Latency, privacy, and knowing when to stop

Crop what doesn't need to be there (other rooms, faces, screens, addresses, reflections), and set retention separately for originals, thumbnails, OCR text, embeddings, and debug captures. Running local cuts what leaves the building, not what an unencrypted disk or an overbroad dashboard account leaks.

Test hosted paths on upload time, regional routing, retention, and rate limits. Test local paths on cold-load time, projector memory, GPU placement, and concurrency. Measure the whole path at p50 and p95, not just generation tokens per second.

Stop tuning once the eval gate passes at acceptable latency, cost, and abstention rate. A bigger model and more pixels are not automatically safer. They mean more private detail exposed and more surface for answering questions the evidence never supported.

Sample for the event, not the clock. That's the rule I'd keep if I had to throw out everything else in this piece.