The OpenAI-compatible endpoint is Ollama's best feature

In February I had a script I didn't want to rewrite. Ticket triage for a client: a few hundred lines wrapped around the OpenAI SDK, classifying inbound tickets into eleven categories. The cloud bill wasn't the problem; the tickets were full of customer data, and the client's legal review for external processing had been stuck for six weeks. On a Tuesday night I pointed the script at the 3090 box in my office instead.

client = OpenAI(
    base_url="http://192.168.1.40:11434/v1",
    api_key="ollama",  # any non-empty string
)

That, plus swapping the model name for a local tag, was the entire migration. I ran our 58-ticket eval set expecting a crater and got agreement within two points of the cloud run.

Working on the first try made me suspicious, so I spent the rest of the evening hunting for what didn't.

what maps cleanly

Chat completions, streaming, tool calling on models that support it, JSON mode. The SDK has no idea it left the cloud. Retries and function-call plumbing all behave the same. That's the actual feature: the /v1 endpoint turns local models from a separate integration into a base URL. Every OpenAI-shaped tool you already run becomes a local-model client for free, which is the same reason swapping providers under LangChain is a config change rather than a rewrite.

It's also the load-bearing wall under the local-first cascade. Try the local tier, escalate to a frontier model when confidence drops. That router is only cheap to build because both tiers speak the same dialect. If local inference needed its own client library, I don't think anyone would bother writing the router at all.

the top_p that did nothing

Now the war story. In March I was tuning the classifier's sampling and swept top_p from 0.2 to 1.0 at a fixed temperature, expecting the label distribution to tighten at the low end. Five sweeps, 380 tickets each. The distributions came back identical. Not similar. Identical, to the point where I assumed my harness was caching responses, and I spent over an hour tearing apart my own code before I thought to question the server.

The build of Ollama I was running at the time accepted top_p on the /v1 path and dropped it on the floor. HTTP 200, plausible completions, parameter silently absent from the sampler. That specific gap was fixed long ago as far as I can tell, but the class of bug is permanent: a compatibility endpoint validates the shape of your request, never the semantics.

An API that accepts your parameter and ignores it is strictly worse than one that rejects it.

My rule since then: every parameter I actually depend on gets a thirty-second canary. Set max_tokens to 7 and confirm the response truncates. Set a seed twice and diff the outputs. Ugly and manual, and it has caught two more silent mismatches since (once after an Ollama upgrade, once with a different runner entirely).

a dialect, not a standard

Other places the mask slips. Error payloads aren't shaped quite like OpenAI's, so error-handling code that inspects structured error types needs its own branch. Model names are your local tags, so any config that hardcodes cloud model ids grows a mapping layer. Newer OpenAI API surface arrives late or never, because a compatibility layer is a chasing target by definition. And the api_key is decorative: there's no auth at all, so the moment you bind beyond localhost you need a reverse proxy and some firewall discipline. Mine is LAN-only, and I still feel mildly guilty about it.

I accept every one of those trade-offs, because the alternative is maintaining two client stacks forever.

The triage script has now been running locally for about five months and somewhere north of 40,000 tickets. The server side of my setup is written up in ollama-in-practice if you want the operational details. The position I'd defend in an argument: forget the model library, the best thing Ollama ships is a boring base URL that makes code you already trust bilingual.