The two-line provider swap is real (mostly)

In February a client asked the reasonable question every client eventually asks: how much of this invoice-extraction bill could we kill by running a local model for the easy documents? Answering it meant swapping the pipeline's model from a frontier provider to a local Ollama endpoint on the 3090 box, and LangChain sells that as a two-line change. It mostly is. The gap between mostly and entirely is where I lost the better part of a week.

the two lines that really are two lines

init_chat_model is the good part, and I want to be fair to it before I start complaining.

model = init_chat_model("ollama:qwen2.5-coder:14b", temperature=0)

Swap that string for the frontier model id and nothing else in the pipeline moves: not the prompt templates, not the tool bindings, not the with_structured_output call, not the LCEL wiring. That is not a brochure promise. Chat calls, tool calling, and structured output all ran through the same interface against a local model speaking the OpenAI-compatible API that Ollama exposes, and my graph nodes never noticed the substitution. For the plumbing, the abstraction is real and it earns its keep.

what the abstraction quietly can't carry

Then extraction quality dropped, and none of it was LangChain's fault.

Sampler params first. My frontier prompt leaned on a temperature and a top_p that meant one thing on that provider; the same numbers on the local model gave mushier output, and the local runner exposed knobs (repeat_penalty, num_ctx) that the frontier API never had. The interface is uniform. The samplers underneath it are not, and the uniform interface is exactly what hides that from you.

Context limits second. The frontier model swallowed a 30k-token batch of invoices without a word of complaint. The local model I'd chosen topped out well below that, and LangChain will happily pass an over-length prompt to a runner that silently truncates the overflow. I lost a full day to extractions that were wrong in a way that looked exactly like a model-quality problem and was actually a context-window problem.

Tone third, and worst. My prompt had been sanded down over months against one specific model's habits: it literally said do not explain your reasoning, because that model liked to editorialize. The local model needed the opposite nudge and a different few-shot example to behave. A prompt is a shape fitted to one model's quirks. You carry it across the swap at your own risk.

The provider abstraction moves your code between models for free, then charges the entire fare on your prompts, which are the part you forgot were provider-shaped.

the discipline that came out of it

So the swap is genuinely free for the wiring and a genuine little project for the prompt, and I now quote a client both numbers. The discipline is the same model-migration reality that any provider change demands: keep a golden set, re-tune the prompt per model instead of assuming it travels, and check the context and sampler assumptions by hand because the tidy interface will never volunteer them.

The cost review landed well, for the record. The local model took roughly 40% of the invoices (the clean single-page ones) at a quality the client signed off on, and the frontier model kept the crumpled scans and the multi-page horrors. That split is now a proper autorouter, which is what the two-line swap had quietly been asking to become the whole time.

Two lines to change the model. Two weeks to change the prompt. Budget for the second number, because it is the real one.