Split the stack: local embeddings, cloud generation (or the reverse)
In March and April I shipped two retrieval systems with exactly opposite architectures. One ran generation on a box in the client's server room and sent embeddings to a cloud API; the other embedded everything locally and sent generation to a frontier model. When I describe this to other engineers, someone always assumes one of the two clients got it wrong.
Both were right.
two clients, opposite answers
Client one is an insurance brokerage. Their retrieval corpus is public product documentation: policy terms and condition sheets anyone can download from the insurers' own sites. Embedding that in the cloud leaks nothing, and it was a bounded batch job: around 60k chunks, embedded once, refreshed quarterly for less than the price of a decent lunch. The queries are the sensitive part (they carry client health details, income, family situations), and the drafted answers even more so. Generation stayed in the building on a 24 GB card, and a mid-size model handles it fine, because grounded summarization over retrieved text is exactly the work the local tier does well.
Client two is a dev-tools company with a monorepo somewhere north of 300k lines. Here the corpus itself is the crown jewels. Their security team had already cleared code snippets flowing to a frontier API under a zero-retention agreement, but nobody had budgeted for embedding-scale traffic. Retrieval means millions of chunks, re-embedded on every push for whatever changed, plus query-time embedding sitting in the interactive path. So embeddings run through Ollama on a workstation GPU, and the couple hundred generation calls a day go to the frontier, where the quality bar for multi-file answers actually gets met.
the matrix I actually use
Three axes, checked per stage rather than per project. Sensitivity: what literally flows through this stage. For embeddings that's the corpus, for generation it's prompts plus finished answers, and the two rarely carry the same clearance. Volume: embedding calls typically outnumber generation calls by two or three orders of magnitude, so a unit cost that looks negligible multiplies into real money on one side and rounding error on the other. Latency: query-time embedding sits where 40 ms versus 300 ms is felt by a human; generation is slow everywhere, so its location barely changes the experience.
There's a fourth axis people skip: the quality gap. Local embedders sit close enough to API ones that retrieval barely notices the difference. Local generators are still visibly behind frontier models on long synthesis. That asymmetry alone explains why the common split is local embeddings with cloud generation. You need a reason, usually sensitivity, to deviate from it.
Sensitivity is a property of a stage, not of a project — the corpus and the prompts rarely have the same clearance level.
where the money actually hides
My budgeting mistake on client two: I priced cloud embeddings assuming the corpus was a one-time cost, and by that math the cloud looked fine. Then tuning started. We re-embedded the entire monorepo 14 times in nine days: 512-token chunks, then 256, then function-boundary chunking, then a second embedder to compare against. Locally that was a warm GPU running overnight. On a metered API it would have been a genuinely uncomfortable invoice and a week of rate-limit queues. The steady state never hurts you; the iteration loop does, and iteration is where all the retrieval quality comes from. The whole experience slotted straight into the 99% cost argument.
the week I forced it local
I owe client two an admission: I spent the first week trying to keep their generation local too, mostly on principle. A 32B coder produced answers that were correct and shallow. Reviewers kept flagging multi-file explanations that stopped one hop short of useful. After seven days of eval sheets refusing to improve, I walked it back. Ideology lost to the spreadsheet, as it should.
The hybrids have real costs too, to be clear. Local embeddings weld your index to one embedder, so swapping models later means re-embedding everything. Cloud generation means the flagship feature inherits someone else's uptime and deprecation calendar. I accepted both trade-offs knowingly, and I'd accept them again.
A split stack feels less pure than either extreme. Purity was never a requirement.