When the model is down: designing AI features that degrade instead of die
The provider has an incident. Their status page goes yellow, then red. And across your product, every feature that calls their API starts throwing errors at the same moment, because each one is a naked call to a single endpoint with no plan for what happens when the endpoint isn't there. Their outage just became your outage, propagated everywhere you used them, and there's nothing your on-call can do but wait for someone else's fix.
This is avoidable, and the way to avoid it starts with naming what a hosted model actually is. It's an external service with an availability you don't control, rate limits you will hit, occasional overload, and the odd full outage. That's not a knock on any provider. It's true of every external dependency, and you'd never build a critical path around a third-party API without handling its failures. The model deserves the same treatment, and it usually doesn't get it, because in the demo the API was always up.
The failures you actually have to handle
There's a short list, and each one wants a response rather than an unhandled exception. Rate limiting, when you've exceeded the requests or tokens per minute the provider allows. Overload, when the provider itself is saturated and asks you to back off. Timeouts and ordinary server errors. And the one people forget because it's specific to models: a refusal, where the call succeeds but the model declines to answer, which your code will mishandle if it assumes every successful response has usable content.
The transient ones, rate limits, overload, timeouts, server errors, want retries with exponential backoff and jitter, so you wait a little and try again rather than hammering a struggling service or giving up on a blip. The SDKs handle some of this for you, and it's worth knowing exactly how much yours does before you assume it's covered. Whatever you do, bound the retries, because retrying forever is its own outage.
A second provider is a hedge, not a luxury
Retries handle a hiccup. They don't handle the provider being down for an hour. For that you want a fallback: when the primary model is unavailable or rate-limited, route to a different model, ideally a different provider, and keep serving. This is one of the strongest arguments for sending everything through a gateway and keeping providers abstracted, because the fallback logic lives in one place and a switch is a routing decision rather than a code change. A second provider costs you the integration work and buys you immunity from the first one's worst day, and at some point of scale that trade is obviously worth it.
When a provider is clearly down rather than flaky, stop sending it traffic at all. A circuit breaker trips after enough consecutive failures, routes everyone to the fallback path immediately instead of making each request wait out its own doomed timeout, and periodically probes to see if the service came back. Without it, a provider outage means every user sits through the full timeout before failing, which turns a backend problem into a miserable frontend one.
Decide what "degraded" looks like, on purpose
Here's the part that separates a resilient feature from a fragile one: deciding, in advance, what the user sees when the AI genuinely can't run. The wrong answer is a stack trace or an infinite spinner. The right answer is a degraded mode you designed, a cached previous result, a simpler non-AI heuristic, an honest "this is temporarily unavailable, try again shortly," or a handoff to a human. The mindset shift is to treat the AI as an enhancement with a fallback rather than a single point of failure, so that when it's down the product gets worse instead of breaking.
A couple of structural pieces support all this. At scale you will hit the provider's rate limits, so you need backpressure, a queue that smooths bursts and sheds load gracefully rather than failing requests at random, and per-tenant limits so one customer's spike can't drain the shared quota and take everyone down. And work that isn't interactive has an escape hatch the interactive path doesn't: a background job on a queue can simply ride out a transient outage with retries, finishing a few minutes late instead of failing, which is often perfectly acceptable for anything the user isn't watching in real time.
The throughline is one sentence. An AI feature built as a single, unguarded, synchronous call to one provider has quietly adopted that provider's worst day as its own SLA. Build it instead as what it is, a dependency on an unreliable external service, with retries for the blips, a fallback for the outages, a circuit breaker so failures are fast instead of agonizing, and a degraded mode you chose rather than one the stack trace chose for you. Then the next time a provider goes red, your users get a graceful "try again in a moment," and you get to watch someone else's incident from the outside.