Put a gateway in front of your LLM calls
Most codebases I've seen pick up LLM calls the way a coat picks up lint. One feature calls the provider SDK directly, then another does, then a third, each constructing its own request, each with the API key pulled from the environment, each handling errors slightly differently or not at all. It works. It keeps working until the day you need to do something to all of them at once, and then you're grepping for SDK imports across thirty files.
The fix is old and boring and I'd recommend it to almost anyone past a toy: route every LLM call through one thin internal layer instead of calling the provider directly from everywhere. Call it a gateway. It can start as a single module and grow into a service, but the principle stays the same. There's one place your code talks to a model, and everything you'd ever want to do to model calls in general, you do there.
Once that chokepoint exists, a surprising amount of stuff that was hard becomes a one-line change.
Provider abstraction is the first payoff, and on its own it might justify the whole thing. When the model you depend on gets deprecated, or a provider has an outage, or a model gets pulled out from under you for reasons that have nothing to do with you, you change one line in the gateway instead of touching every call site. Your application code asks for "the model" and the gateway decides which actual model that is today.
That decision can be smart. Routing by difficulty or cost lives naturally in the gateway: the easy requests go to a cheap model, the hard ones to a frontier model, and the whole cascade is implemented once, centrally, instead of reinvented in every feature. Fallback lives there too. When the primary provider returns an error or a refusal, the gateway can fail over to a backup instead of surfacing the failure to the user. A provider's bad day turns into a non-event for your app.
Caching is cleaner in one place. Beyond the prompt caching the provider does, you can add response caching at the gateway for repeated identical requests, and the logic lives in exactly one spot instead of being copy-pasted or, more likely, forgotten. Same story for rate limiting and quotas. A runaway loop or an abusive user can run up a frightening bill fast, and the gateway is where you put per-feature and per-tenant budgets that cap the damage before it reaches your invoice.
Then there's observability, which I'd argue is reason enough by itself. Every model call passing through one point means every call can be logged, traced, and cost-attributed without instrumenting each feature separately. When someone asks why the LLM spend tripled last week, you have an answer, because it all went through the place that counts. And in the worst case, that chokepoint is a kill switch. A feature misbehaving in production can be throttled or turned off centrally, immediately, instead of requiring a deploy to every service that happened to call the model.
A small but real bonus: the provider credentials live in the gateway, not sprinkled through application code where they leak into logs and git history. One service holds the keys. The rest of your code holds nothing sensitive.
None of this is free. The gateway is another thing to run and another network hop of latency, and if you over-build it on day one you've added a distributed-systems problem to a feature that never needed one. So the honest scope: a single-script prototype or one call site doesn't want a gateway, it wants you to just call the SDK and move on. The gateway earns its place once you have several call sites, or more than one model in play, or production-grade concerns about cost and reliability, which is to say, once you have something real. And it can grow with you. The first version is a function everything goes through. You reach for a full proxy service, your own or one of the open ones, when scale demands it.
The mental shift underneath all of it is to stop treating an LLM call as a library function. Start treating it as what it is: a call to an external service with cost, latency, rate limits, and a non-trivial failure rate. You would never let thirty parts of your codebase each independently call a payment provider with no shared client, no retry policy, no central logging, and the secret key inlined. Model providers are the same kind of dependency. Give them the same front door.