Knowledge graphs vs vector RAG: when relationships beat similarity

Vector RAG answers one kind of question extremely well: "find me text that's about this." Embed the query, pull the most similar chunks, generate. But a whole class of real questions isn't about similarity at all — it's about relationships, and on those, vector search quietly fails. That's the gap knowledge graphs fill.

Where vector similarity hits a wall

Similarity retrieves chunks that resemble the query. It has no notion of how chunks connect. So it struggles with:

• Multi-hop questions. "Which services depend on the component that owns the orders table?" needs traversal: table → owner → dependents. No single chunk says all of that, and the chunks that do exist don't resemble the question.
• Aggregation across documents. "What are all the decisions that touched authentication?" needs to gather scattered, differently-worded references and relate them — not return the top-5 most auth-similar paragraphs.
• Global structure. "Summarize the themes across this whole corpus" is a question about the shape of the data, which similarity-over-chunks can't see.

The tell is the same every time: the answer lives in the connections between pieces of content, and similarity only sees the content.

What a knowledge graph adds

A knowledge graph stores entities and the relationships between them — OrderService —[owns]→ orders_table, PaymentService —[depends_on]→ OrderService. Now you can query the connections: traverse from a node, follow edges, answer multi-hop questions by walking the graph instead of hoping a chunk happens to contain the whole chain.

GraphRAG is the pattern that marries this to LLMs: extract a graph from your corpus, and at query time retrieve a relevant subgraph (entities + their relationships) as context, instead of (or alongside) flat text chunks. The model reasons over structure, not just resemblance.

Vectors find what looks like the answer. Graphs find what's connected to the answer. Multi-hop and "how does this relate to that" questions are the graph's home turf.

The honest tradeoff

Graphs are not a free upgrade:

• Building the graph is expensive. Entity and relationship extraction is real work, often itself an LLM pipeline, and it has to be maintained as the corpus changes. A stale graph is a confidently-wrong graph.
• Vectors are cheap and fast. An embedding index is trivial next to a curated graph, and for "find text about X" it's strictly better — simpler and just as accurate.
• Most questions don't need traversal. The majority of real queries are content-similarity questions that vectors nail. You'd be paying graph-construction cost to answer questions a reranker already handles.

The hybrid that actually wins

The production answer is rarely either/or — it's route by question type:

• Vector + reranker for content recall — "find the relevant text." The default, and the rag-that-retrieves stack covers it.
• Graph traversal for relationship and multi-hop questions — "what connects to what."
• A router (often the model itself) decides which the question needs, or runs both and merges.

The lazy rule

Do not build a knowledge graph because it's impressive. Build it when you have real, recurring multi-hop questions that vector RAG demonstrably fails on — and not before. Start with vector + reranking; instrument what it gets wrong; if the failures cluster around relationships and traversal, then a graph earns its construction cost. Otherwise it's infrastructure you built six months early and now maintain forever. The graph is a powerful tool precisely because it's a specific answer to a specific failure — reach for it when you've seen that failure, not when you've read about the tool.

Inspired by the RAG and knowledge-graph writing at tomaskubica.cz.