Cohere Embed 5: index with the big model, query with the fast one

Embedding models are the least glamorous part of a retrieval pipeline and, in my experience, the part that decides whether the pipeline works. Cohere released Embed 5 on September 30, and what caught my eye is not the benchmark table. It is a design choice that goes straight at a cost problem every RAG team eventually meets.

Here are the facts as Cohere states them. Embed 5 comes in two variants. Pro costs $0.12 per million tokens, Fast costs $0.08 and offers about 2.4 times the throughput. Both support a 128K context, more than a hundred languages, and multimodal input. The dimensions are Matryoshka style, so you can truncate the vectors at 2048, 1536, 1024, 768, 512 or 256. Cohere reports that Pro leads on ViDoRe V3 with 85.8, and on finance tasks with 80.1 on FinanceBench and 90.0 on FinQA. Those are vendor-reported figures and I have not reproduced them.

The asymmetry is the idea

The key sentence in the announcement is that Pro and Fast share an embedding space. That means a document vector made by Pro and a query vector made by Fast can be compared directly. In practice you can index your corpus once with the expensive, higher quality model and answer user queries with the cheaper, faster one, without re-indexing.

This matches how the cost actually falls in a production system. Indexing is a batch job. You pay it once per document, you can run it overnight, and quality matters a lot because a poor vector is baked into the index until the next rebuild. Querying is the opposite: it happens on every user request, latency is visible to a human, and the volume grows with your product. Spending more where it is rare and less where it is constant is just sensible, and it is surprising how many pipelines use the same model for both because that was the default in the tutorial. If you are still choosing, our notes on picking an embedding model cover the basics.

Matryoshka is a dial, not a free lunch

The second lever is dimension. A Matryoshka-trained embedding is arranged so that the first N coordinates still carry a usable representation. Cut a 2048 dimensional vector to 256 and you keep a coarser but real signal. The storage and search savings are direct: an eightfold reduction in vector size means roughly an eighth of the memory for the index and faster similarity search.

What you give up is recall on the hard cases, the queries where two documents are nearly identical in topic and differ in a small detail. The loss is usually small on easy corpora and larger on dense, specialized ones such as legal or financial text. The honest way to know is to measure it on your data, and I would do that before anyone commits to a number.

A test I would run before switching

I would take a few hundred real queries with known relevant documents, even a rough set labeled by a colleague. Then I would build four indexes: Pro at full dimension, Pro at 512, Pro at 256, and perhaps your current model as a baseline. For each, I would query with both Pro and Fast and compare recall at the cutoff you actually use, usually the top five or ten results. What you are looking for is the cheapest combination whose recall drop you can live with.

Two further checks matter more than people expect. Measure the latency of your whole query path, not the embedding call alone, because the vector database and the reranker may dominate. And look at the failure cases by hand. A recall number tells you how often the right chunk is missing; reading ten misses tells you why, and the answer is often chunking, not the model. The ingestion pipeline notes go into that side.

The cheapest embedding is the one you do not have to compute twice.

Where this fits next to local options

If your data cannot leave your network, the hosted route is closed and the same ideas apply to open models you run yourself. The shared space trick is not unique to Cohere in principle, but it is unusual to get it as a supported feature, so you would have to verify it for any local pair. I covered the self-hosted side in running embeddings locally with Ollama, and the same truncation experiment works there.

What I do not know

I cannot tell you how well the shared space holds under a domain shift, for example when the queries are short keyword fragments and the documents are long reports. Cohere's numbers come from standard benchmarks, which behave differently from messy enterprise text. I also do not know how the pricing compares once you count the cost of re-embedding after a future model change, which is the real long-term bill for any embedding choice.

My practical read is simple. If you run a RAG system with real traffic and you are paying for the same large model on every query, the asymmetric setup is worth a day of testing. If you are early and your corpus is small, ignore all of this, pick something reasonable, and spend the time on your chunks. The cost curve only starts to matter once people use the thing.