Agentic RAG: when retrieval becomes a tool the agent drives

Classic RAG is a pipeline: embed the user's query, retrieve the top-k chunks once, stuff them in the prompt, answer. One shot. It works well for "find me text about X," and falls apart the moment the question needs more than one look. Agentic RAG fixes that. It hands retrieval to the model as a tool the model drives in a loop.

Where one-shot retrieval breaks

The single-retrieval assumption fails in predictable ways:

• The user's words aren't the right search query. A human asks "why is checkout slow?"; the relevant code says nothing about "slow." One-shot retrieval searches the literal question and misses.
• The answer needs several lookups. "Which services would break if I change this schema?" requires finding the table, then its owner, then the dependents. Three searches, each informed by the last.
• The first results reveal what to search next. You don't know the right second query until you've read the first results. A pipeline can't react. A loop can.

These are the multi-hop and reformulation cases, and they're common.

What agentic RAG does

Give the agent a search tool and let it drive retrieval: decide what to query, read the results, reformulate, search again, and stop when it has enough, or admit it doesn't. Retrieval stops being a fixed step before the model. It becomes part of the model's reasoning loop.

Here's what that gets you:

• Query reformulation. The model writes better search queries than the user's raw question, and rewrites them based on what came back.
• Iterative, multi-step retrieval. Search, learn, search again, following the trail.
• Knowing when to stop. Enough evidence to answer, or enough dead ends to say "I can't find it" instead of hallucinating.

Classic RAG asks one question and trusts the answer. Agentic RAG investigates: it searches, reads, and searches again, the way a person actually looks something up.

The cost, named honestly

Agentic RAG means more model calls, more latency, more tokens. Each retrieval round is another loop iteration. One-shot RAG is cheaper and, for the queries it handles, just as good. So it's a tool for the queries that one-shot demonstrably can't answer, not a blanket upgrade you apply everywhere.

The natural pattern is the cascade applied to retrieval: cheap one-shot retrieval for the easy queries, escalate to the agentic loop only when one shot fails (no good results, or the question is obviously multi-hop).

Building it

• Search as a tool. Expose retrieval as a function the agent calls, directly or via MCP. The agent loop does the rest.
• Bound the iterations. A budget or turn cap so it doesn't loop forever on an unanswerable query.
• Format tool results tightly. Don't dump 20 raw chunks back into context each round. Summarize and paginate, or you poison the window and the cost explodes.
• Keep the retrieval fundamentals. Hybrid search and reranking still matter inside each round. Agentic RAG drives better retrieval; it doesn't excuse bad retrieval.

The lazy rule

Start with one-shot RAG plus reranking. It's cheaper, simpler, and handles most queries. Add the agentic loop only when you have real queries it provably can't answer: multi-hop questions, ones that need reformulation, investigations. Measure where one-shot fails, and let that evidence, not the coolness of the idea, pull you up to the loop. Agentic RAG earns its keep because it targets one-shot's specific failures. It isn't a default (the ponytail rule, again).