Upgrading your model is not a one-line change

A new model comes out, it tops the leaderboard your old one was on, and the upgrade looks like the easiest win of the quarter. Change the model ID, redeploy, enjoy the free quality bump. Except a week later support is forwarding you complaints, and three behaviors that worked perfectly on the old model are now subtly broken, and you're git-blaming a one-character change wondering what you did.

What you did was treat a migration like an edit. Swapping the model string is the smallest part of changing models, and the fact that it's a single character hides how much actually moved underneath it.

The new model is a different distribution

Your prompts weren't written in a vacuum. They were tuned, consciously or not, against one specific model's quirks. Every "IMPORTANT: you MUST always use the search tool" was you compensating for that model's reluctance to search. Every careful bit of phrasing was shaped by how that model responded. A new model has different reluctances and different defaults, and your old compensations are now miscalibrated against a model that didn't need them.

The classic version of this: a model that used to under-trigger a tool gets replaced by one that follows instructions more faithfully, and suddenly all those forceful "you MUST" instructions cause it to over-trigger, calling the tool constantly when it shouldn't. The instruction stayed the same, but the model's relationship to it changed. Newer models also tend to be more literal, generalizing less from one example to the next, so a prompt that relied on the old model "getting the idea" from a couple of cases now does exactly and only what you literally wrote. Verbosity shifts. One model is terse where the last was chatty, or the reverse, and your UI that assumed a certain response length now looks wrong. Some models start asking the user more clarifying questions; some ask fewer and just proceed. None of this throws an error. It just changes the product's behavior in ways your users feel before you do.

The API surface can break under you too

Beyond behavior, the contract can change. Newer models drop parameters that older ones accepted, so the sampling settings or the thinking-budget configuration you were passing now returns a 400 instead of being quietly honored. Response prefilling that you relied on to force an output format may no longer be allowed. New stop reasons appear that your code doesn't handle, so a perfectly normal response falls through your branches. And the tokenizer can change between model generations, which means the same text counts as a different number of tokens, which means your carefully-set max output limit and your cost projections are all silently off until you re-measure.

There's also the cache. Switching models invalidates your prompt cache entirely, so the first wave of requests on the new model pays full price to warm a cache that was saving you 80% a moment ago. If you don't expect it, the cost graph spikes and you panic over nothing.

You don't get to opt out forever

It's tempting, after reading all that, to just never upgrade. Pin the old model and leave it. That works until it doesn't, because providers retire models on their own schedule, and a deprecation notice eventually becomes a hard cutoff. The model you froze your stack on stops answering one day, and now you're doing the migration you've been avoiding, except under time pressure and probably during an incident. Building on a hosted model means accepting that migration is a recurring tax you keep paying, which is one more argument for keeping your provider abstracted so a swap is a config change rather than a rewrite.

Treat it like the migration it is

The fix takes more work than the one-character diff suggests. Run a representative sample of your real traffic through the new model and read the outputs before you ship to anyone. Run your eval set on both the old and new model and compare; if you don't have an eval set, this is the moment you'll wish you did, because it's the only way to see a regression that isn't the one a user happened to report. Re-tune the prompts rather than assuming they transfer, paying special attention to any forceful instructions that might now overtrigger. Re-baseline your token counts and cost. And change one thing at a time: don't swap the model and rewrite the prompts in the same deploy, or you'll have no idea which change caused which effect.

The counterintuitive thing to sit with is that a better model can make your application worse, at least at first, because your scaffolding was overfit to the old one. Better on a generic benchmark is not the same as better inside your specific harness with your specific prompts. The benchmark measures the model. Your evals measure your product. Upgrade against the second number, not the first, and the new model's gains will actually reach your users instead of getting eaten by the regressions you didn't look for.