Testing LangChain apps without burning tokens

For about three months my CI ran real model calls on every push. I only found out what that cost when the client's finance person flagged a line item: the test suite alone was burning somewhere around 90 dollars a month in tokens, most of it re-answering the same ticket-triage fixtures on every commit to what was basically a two-person project. That was the week I built a real test pyramid for the LangChain pipeline, and the week I learned exactly what it would not protect me from.

the shape I landed on

Bottom to top, it goes like this:

• Unit tests with a fake model for all the logic: routing, state updates, the retry path, the graph edges.
• Recorded fixtures for the parsers: real model output captured once and replayed forever after.
• A tiny smoke set of real calls, maybe a dozen, that runs only on merge to main.
• The eval suite, which runs nightly and grades quality rather than correctness.

Ninety-odd percent of the tests touch no network at all.

The token bill fell to a rounding error, and CI dropped from around ninety seconds to under ten. That alone would have justified the afternoon it took.

FakeListChatModel carries more than you'd expect

The workhorse at the bottom is FakeListChatModel. You hand it a list of strings, it returns them in order, and it stands in for the real model everywhere your logic runs.

from langchain_core.language_models import FakeListChatModel
model = FakeListChatModel(responses=['{"category": "billing", "priority": 2}'])
chain = prompt | model | parser

That is enough to test everything that is not the model's judgment: does the parser accept this shape, does the router send a priority-2 billing ticket to the right queue, does a malformed response actually trigger the retry instead of blowing up. I test the retry by feeding a bad string followed by a good one and asserting recovery. All of that is my code, and my code is where most of my bugs genuinely live. The wider argument for testing AI systems sits underneath this; the fake model is just how you act on it cheaply.

the regression every mock waved through

Now the confession. In April I tightened the triage prompt (one added sentence, meant to fix a specific misclassification). Every mocked test passed, because mocks return canned strings and canned strings could not care less what your prompt says. The smoke set passed too; its dozen tickets happened to miss the affected class entirely.

That one sentence had quietly nudged the model toward marking ambiguous tickets as urgent. It shipped clean. Priority inflation crept in over about a week, until roughly a third of the client's urgent queue was false alarms and the genuinely urgent tickets were buried under them.

A mock proves your code does what you wrote; only a real model on real examples proves your prompt does what you meant.

What eventually caught it was the nightly eval suite graded by an LLM judge, which tracks urgent-precision as a number, and the number fell off a cliff the night after I shipped. That is the division of labor I trust now. Mocks guard the code, evals guard the prompt, and those are genuinely different jobs. Mocks are instant and blind to meaning; evals are slow and the only thing in the stack that can see it.

If your app is a graph, the same split holds with extra care around testing the graph itself: the edges mock cleanly, the node contents do not. Mock what you wrote. Eval what you meant. Pay for real tokens exactly where meaning lives, and not one dollar anywhere else.