Gemini 3.5 Flash: An evaluation set worth keeping
A tool-calling endpoint fails at the volume it runs, not in the one demo conversation that got it approved.
Gemini 3.5 Flash showed up on the usual leaderboard cycle, tempting a start from the score and working backward. Google lists it as a current stable model, pointing new projects at the GA Interactions API: a fast multimodal model for high-volume interactive and tool-assisted work, testable under gemini-3.5-flash. Prices, aliases, and availability shift, so check Google's docs before committing to production.
Fluency isn't the bar you're testing
Fluency isn't correctness. Define pass before the first prompt: a schema-valid record tracing to source for extraction, a patch that clears tests and review for coding, a claim set with checkable sources for research.
The method: pull real failures, strip anything sensitive, write what a correct answer must satisfy, score blind, repeat after every model or prompt change. Keep the whole request: system instructions, tool schemas, files, sampling settings, output caps. Native tools versus pasted terminal output means different tasks, not different prompts; hold the workflow steady, format aside.
| Log this | Why |
|---|---|
| Model ID, dated API version | what ran |
| Prompt, tool-schema revision | silent drift |
| Input, cached, reasoning, output tokens | cost vs capability |
| Queue, first-token, total time | where latency lives |
| Pass, retry, abstain, human repair | real success rate |
| Provider error, fallback route | failure behavior |
The failures you keep matter more than the wins
The costly mistake: an easy test set that confirms the purchase and never catches a regression. Frontier output is plausible by default, hard to catch. Measure whatever ordinary code can check, have a reviewer flag concrete defects, and keep the wrong answers. They end up defining your routing rules and guardrails; a folder of wins won't.
Long context earns more suspicion, not less, as long-context workflows keeps arguing. A bigger window doesn't remove prefill time, retrieval noise, or the line between trusted instructions and untrusted documents. Cache stable prefixes first, keep volatile metadata off the cache boundary, send a structured checkpoint instead of a full history. The same restraint applies on output: a short answer with evidence beats an unsolicited tutorial.
Better function selection doesn't relieve your app of validating arguments, authorizing the caller, capping side effects, deduplicating requests, or logging what happened. A web page, a file, an email, a tool result: data, not authority. A smarter model is no substitute for a narrower interface.
Pin the version, keep a way out
Promote a model once it clears task-specific gates and remaining failures fail safely; put that in configuration, not habit. Log model identity, phase timings, token categories, and eval outcomes by default; skip sensitive prompt content. Keep a cheap, fast lane for routine work and a clear trigger for hard cases: the strongest model earns its keep on requests that need it, not everything sent. A discount tier stops being a discount once retries and review eat the savings.
Treat the switch as ordinary maintenance: pin dated versions where stability matters, watch deprecation notices, canary new aliases before full rollout, keep one known-good fallback. Re-run the local eval set whenever the model, prompt, tool contract, or reasoning default changes.
None of this is free, and if your team hasn't got enough real failures to build a corpus from, the framework is aspirational until you do.