Embeddings for code search: why your semantic search misses the obvious

Semantic code search, and any RAG over a codebase, runs on embeddings: map code to vectors, find the nearest neighbors to your query. It's powerful, and it fails on code in ways it never fails on prose. If your "smart" code search keeps surfacing the wrong function, here's why, and how to fix it.

A 20-second recap of embeddings

An embedding model maps text to a vector such that similar meaning lands at nearby points. Search becomes nearest-neighbor lookup: embed the query, find the closest chunks. Great for "find text about authentication." (More in RAG that retrieves.)

Why code breaks the assumptions

Code isn't prose, and the differences bite:

• Exact symbols matter, and embeddings smear them. "Where is OrderService.cancel called?" is an exact-match question. Embeddings dissolve it into "stuff about cancelling orders" and rank a vaguely-related function above the literal call site. Vectors are structurally bad at exact identifiers.
• Meaning lives in structure, not description. A function's behavior is in its calls and control flow, not in an English summary. Embedding the raw text captures the surface and loses the structure.
• Boilerplate looks similar to everything. Getters, imports, error-handling scaffolding: huge swaths of code are surface-similar, so similarity ranking buries the relevant-but-different result under near-duplicates.
• Relationships are cross-file. Embeddings see each chunk in isolation; "what depends on this?" is a graph question, not a similarity one.

Cosine similarity feels objective on code, and it isn't. It will confidently rank a function that reads like your query above the one that is the answer. Exact-match questions are not similarity questions.

How to do code embeddings well

• Chunk on structure, not line counts. Split on functions and classes, and keep a signature with its body. Guillotining a function mid-body, or orphaning it from its signature, produces a chunk that embeds to noise. This is the fix that pays off most (chunking).
• Use a code-specific embedding model. Models trained on code understand identifiers and structure far better than a general-text embedder. Don't search code with a prose model.
• Go hybrid: vector and keyword. This is the one that matters most. Keyword/BM25 (or plain grep) nails the exact identifiers and error strings that embeddings smear; vectors handle the conceptual "find the retry logic." The union is real code search. Neither alone is.
• Carry metadata and filter on it. File path, symbol name, language. Half of "find the right thing" is filtering before you ever compute a similarity.
• Rerank. Pull a generous candidate set, then reorder with a model that reads query-and-code together (the biggest quality jump per line on code too).

The honest rule

For "find this exact symbol or string," grep wins. It's exact and instant. For "find the code that does X conceptually," embeddings win. Real code search needs both: combine grep with a vector DB rather than replacing one with the other, and you get results that neither approach delivers alone. Run embeddings locally (cheap and private), chunk on structure, pair with exact search, and rerank. Do that and semantic code search stops surfacing the plausible-looking wrong function and starts finding the one you meant.