OpenAI-compatible does not mean behavior-compatible

So you point the same client at a different backend, expecting nothing else to move.

The base URL matches. The schema matches. The behavior doesn't: retries fire wrong, tool calls carry different argument shapes, token counts don't line up, a clean 400 becomes an error you can't parse. Ollama, llama.cpp, vLLM, and hosted endpoints all claim compatibility and still disagree on stream endings and unsupported fields. Wire resemblance isn't a contract.

Compatible on the wire is not the same claim as compatible in production.

Local's edge: every layer stays inspectable, not a vendor's number.

Decide what "better" means before you touch a flag

Name what has to improve before touching a flag: first-token latency, jobs per hour, resident model count, fewer corrections. "Faster" is a mood, not a target. Measure the whole workload: evidence in, answer out, quality gate passed, system still serving others.

Pin everything, then log the phases separately

Pin the model artifact, tokenizer, prompt template, runtime build, launch command, sampling settings, then run against real workload inputs, edge cases included. Run cold once if that's real, then warm long enough for cache and thermal effects to show. Write contract tests per feature, normalize at one gateway, and log phases separately, queue wait through decode rate plus resource peaks. Token speed is diagnostic, not the result: count valid records, review time, or first-token latency, whichever fits. A short, warm win can lose once model swaps and real prompts arrive, per the run-local-llm-guide.

The retry logic is where compatibility breaks

The recurring mistake: swap the base URL, assume retry and parsing behavior travels with it. Local runtimes keep working past where they should stop: offloading layers, paging memory, missing caches, falling back to a generic kernel. That resilience hides a degraded setup. Read the logs; confirm it's active. Change one variable at a time, keep outputs not just numbers: quantization, sampling, and model swaps can make an answer faster and wrong together. Shorter answer, check it's complete; faster extractor, count valid records, not braces. An undocumented-patch gain is a bad trade for anything shared. Boring configurations age well.

Write the boundary down, not just the number

Support the subset your tests prove, fail loud on the rest, and write that into the result file with the workload, date, and reasoning, next to what should force a retest: a new model family, longer contexts, a different traffic mix. Skip that and the numbers turn into folklore. Leave slack past the target: free memory for prompt variance, spare capacity for interactive users, thermal margin so the box runs a week, not an afternoon. Efficiency isn't packing everything to a hundred percent; it's finishing predictably and cheaply.

None of this hands you a contract test that stays true on its own. Every runtime updates its OpenAI-compat layer on its own schedule, and the drift shows up first where nobody's suite looks: how a tool call streams its arguments, what an error body looks like for a new failure. I don't have a clean fix beyond rerunning tests more often than feels reasonable.