The RAG pipeline I actually ship with LangChain

In March I shipped a compliance assistant to a fintech client: roughly 7,300 PDFs and Confluence pages, a few hundred questions a day from their ops team, answers that get pasted into audit responses. It was my third RAG build on LangChain and the first one I'd call production-shaped rather than demo-shaped. This is the pipeline that survived contact with real documents.

the loader zoo is the real product

People argue about LangChain's abstraction tax, and on plain retrieval chains I mostly side with the skeptics. The document loaders are the exception. This client had PDFs from four generations of scanning software, a Confluence wiki with eleven years of sediment, and a CSV register of controls. The Confluence loader alone saved me what I honestly estimate as two sprints of auth-and-pagination plumbing, the kind I've written from scratch twice before and hated both times.

The caveat: loader quality is wildly uneven. I went through three PDF loaders before settling on the PyMuPDF-based one, because the first two mangled tables in ways I only caught by diffing extracted text against the originals for a sample of 40 documents. Budget a day for that diff. It's dull work and it found the problem that mattered most.

the default splitter betrayed me

In the notebook, recursive character splitting looked fine. The chunks were tidy and the retrieval demo sailed.

Two weeks into the pilot, a compliance officer asked why the assistant claimed a certain incident class had a 72-hour internal deadline when her SLA table said 24. The splitter had cut that table in half mid-row: category names in one chunk, hour values in another. The retriever served the half without the numbers next to a paragraph about the general 72-hour rule, and the model stitched them into a confident wrong answer. The model did nothing wrong. Every default I hadn't questioned did.

That one cost me a weekend and some credibility.

The fix wasn't exotic. Everything converts to markdown-ish text first, splits on headings so sections stay whole, then gets size-capped. After testing five configurations against our golden questions I landed near 700-token chunks with 80 tokens of overlap, and tables stay atomic whatever their size. Your numbers will differ. The point is that splitting is a modeling decision, and a default is somebody else's model of your documents.

embeddings: the boring middle

Embeddings were the least dramatic choice in the stack. We use an unremarkable hosted embedding model, batched at ingest. Swapping it later means re-embedding the corpus, and at this size that's an overnight job and a small invoice, not a crisis. The decision that actually moved retrieval quality lived upstream in the splitter, not in which vectors we bought.

retrieval: mmr, and a k you have to earn

Plain similarity search failed in a way I should have predicted: compliance docs live in near-identical versions, so the top four hits were routinely the same clause from three different years. Maximal marginal relevance fixed the redundancy, for the price of two parameters nobody can intuit up front.

retriever = store.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 6, "fetch_k": 30, "lambda_mult": 0.6},
)

The client's team wrote 120 golden questions with known source passages. On that set (our set, not a benchmark), the share of answers with usable context went from roughly 60% under default similarity search to the high 80s after MMR plus the splitter rework. I stopped tuning when a full week of fiddling bought two more points. Most of the remaining misses are questions where single-shot retrieval is the wrong shape entirely, and no value of k rescues those.

citations or it doesn't ship

Every chunk carries its document id and section path in metadata. The prompt requires the model to name the ids it used, the UI renders them as links into the source PDF, and when retrieval comes back thin the assistant says so and stops.

A RAG answer without a citation is a hallucination with good posture.

The compliance team trusts the assistant more for its refusals than for its answers, which took me embarrassingly long to accept. The first full ingest, for the record, ran overnight on my Mac Studio before the pipeline moved into the client's VPC.

what I still hand-roll

Three things stay out of the framework, deliberately:

• Reranking: a small cross-encoder pass over the top 30 candidates, about 40 lines of code I understand completely.
• Evals: the golden set runs as plain pytest in CI, and a human reads the failures monthly.
• Ingestion orchestration: a queue, a Makefile, idempotent upserts. Frameworks keep offering to own this and I keep declining.

When single-shot retrieval genuinely can't answer (multi-hop questions, reconciling two contract versions), I reach for agentic retrieval rather than cranking k up to 20 and hoping. And for the client whose documents can't leave the building, the same shape runs fully offline with local embeddings and a local model, slower and a little grumpier.

The pipeline is boring on purpose. Audit-adjacent software should be.