Choosing an embedding model is a decision you'll be stuck with
Most decisions in an LLM stack are easy to walk back. You can swap the model with a config change, trade one vector store for another, rewrite a prompt. The embedding model is different, and people underrate how different. The moment you embed your corpus, the choice is baked into every vector you've stored, and the vectors from one embedding model are not interchangeable with another's. They live in different spaces. Switching embedders means re-embedding everything you have and rebuilding the index. At any real scale that's a project with a budget, not a Tuesday afternoon.
That stickiness is the whole reason to choose carefully up front, and it's the reason the usual selection method, glance at a leaderboard and pick the top one, is a mistake.
The leaderboard tells you about someone else's data
There are good public benchmarks for embedding models, worth a look, but understand what they measure: average performance across a broad set of generic tasks. Your application is not generic. You have a specific domain and a specific vocabulary, and a model that tops the general benchmark can quietly underperform on your legal documents, your codebase, your medical notes, your particular jargon. The leaderboard is a filter for the first cut, narrowing dozens of options to a handful worth testing. It is not the test.
The test is your own data. Build a small retrieval eval the same way you'd build any eval: a set of real queries from your domain, each paired with the documents you know are the right answer, drawn from your actual corpus. Then run your candidate embedders against it and measure how often the right document lands in the top results. This is the only number that tells you anything about how the model will behave in your product, and it routinely disagrees with the leaderboard ranking, because the leaderboard was never about your corpus.
What actually moves the needle
A few properties matter more than the headline score.
Domain fit is the big one. An embedder trained on general web text will smear the exact identifiers and structure that matter in code, and a monolingual model will fail you on a multilingual corpus. Matching the model's training to your content's nature beats a couple of points of generic benchmark.
Dimensions are a tradeoff people treat as "bigger is better." Higher-dimensional embeddings cost more to store and more to search, and the quality gain over a well-chosen smaller model is often marginal. Pay for the dimensions when your eval shows they earn their keep, not by default.
Maximum input length is a quiet failure mode. If the model truncates inputs longer than some limit and your chunks exceed it, you're silently embedding only the first part of each chunk and losing the rest, with no error to warn you. Check that the model can actually ingest your chunk size.
And then the deployment question, which loops back to stickiness. A hosted embedding API is easy to start with, but it's a dependency with a per-call cost and, worse, a model that can change or be deprecated under you, at which point you're forced into the re-embedding project on someone else's timeline. An open embedding model you run yourself is free per call and private, and never deprecated out from under you. Given that switching costs a full re-embed, there's a real argument for preferring an embedder you control, for the same provider-risk reasons that apply to the generation model, except here the lock-in is heavier because the cost of moving is your entire corpus.
This is the sticky choice in the stack, and it earns the care: leaderboard for the shortlist, your own queries and documents for the real test, domain fit weighted above the generic ranking, chunk size checked before you commit. The vector database underneath is swappable any time you like. The embeddings sitting in it are not, at least not cheaply, and that asymmetry is why this decision deserves more care than the ones around it.