Local embeddings: the part of the stack that never left

My generation traffic has spent two years drifting to cloud models. Long agentic work needs frontier reasoning, and I've stopped pretending otherwise. One workload never left the 3090 box, and I don't expect it ever will: embeddings.

The setup is unremarkable on purpose. A code-search index over a client monorepo plus my own repos (about 210,000 chunks), embedded with a small open model through Ollama's embeddings endpoint and stored in pgvector on a Postgres that was already running anyway. A nightly job re-embeds whatever changed; most nights that's a few hundred chunks, finished before the kettle boils. Queries embed locally in a handful of milliseconds, retrieval stays on the box, and the answer synthesis on top can go to whatever model deserves it. The retrieval half of this is written up in embeddings-for-code-search.

The economics are lopsided in a way generation's never were. Embedding is one forward pass through a model a fraction of the size of anything I'd chat with; a full rebuild of all 210k chunks takes under two hours on the 3090, and the nightly delta is seconds. Cloud embedding APIs aren't expensive: they're just not buying me anything here, and for client code the privacy property ends the conversation before price even comes up.

Privacy closed the argument; the low cost just made it comfortable.

the burn

Last autumn a leaderboard convinced me a newer embedding model was better, and I believed it on a Wednesday. I pointed the nightly job at the new model and didn't re-embed the existing index. The dimensions happened to match, so nothing errored, and cosine similarity cheerfully compared vectors from two unrelated spaces and returned confident noise. Fresh code became nearly invisible to search. I only caught it four days later, when a function I'd written that same morning ranked forty-something for a query containing its own name.

Vectors from two different models don't disagree; they don't even share a language.

The repair was a full wipe and an overnight re-embed, which locally is a rounding error on the electricity bill. The lesson was structural: the embedding model is a schema decision wearing a config knob's clothing. Mine is now pinned in the index metadata, and the ingest job refuses to write if the model tag doesn't match. Choosing an embedding model deserves real deliberation precisely because you'll pay to change your mind later.

the honest caveat

Local open embedders still trail the best proprietary ones on hard retrieval benchmarks, and the long-context options are thinner. For code chunks of thirty to sixty lines, I couldn't measure the difference in my own hit rates, so I stopped worrying about it. But if your retrieval problem is subtle, test before you commit. And if you want the generation side offline too, local-rag-fully-offline covers the whole pipeline, not just this half.

Generation will keep sloshing between local and cloud with every release cycle. The index stays home.