Test-time tool evolution: when the agent should write its own tools instead of calling MCP

An arXiv paper that surfaced in early August, via the VoltAgent list of 2026 agent papers, proposes what its authors call test-time tool evolution: instead of handing the agent a static library of tools, the agent synthesizes executable tools during inference, verifies them, and evolves them as the task proceeds. The target domain is scientific reasoning, where the right tool for a step often does not exist until you are looking at the data. A companion paper, JudgeFlow, works the other end of the loop with a "block judge" that scores how much responsibility each block of an agent's logic bears for the outcome, feeding an eval-judge-optimize cycle.

What the papers propose

Tool evolution moves the boundary between tool calling and code execution. Today, in the MCP model, a server publishes a fixed set of tools with typed inputs, the model picks one, and the runtime executes it. The tool's code was written by a human, reviewed, and deployed. In the evolved-tool model the agent writes the tool, verifies it, and keeps or mutates it based on whether it helped. The library is an output of the run, not an input.

JudgeFlow is more modest and more immediately useful. A multi-step run fails and nobody knows which step was wrong; a judge that assigns responsibility per logic block turns "the run failed" into "block four is where it went wrong", which is the precondition for optimizing anything.

When dynamic tools are worth it

The cases where I would let an agent write its own tools are narrow.

• The task space is open-ended and a wrong answer is cheap. Exploratory analysis, hypothesis generation, throwaway scripts.
• Verification is cheap and automatic. If the synthesized tool can be checked against ground truth (a unit test, a held-out dataset, a physical constraint), evolution has something to select on. Without that signal, evolution is a random walk.
• No tool exists and writing one by hand is slower than the task. The scientific framing is exactly this: a twenty-line function nobody has written yet.

A tool the agent wrote at 2 a.m. is code nobody reviewed, running with the agent's permissions, and that is a security model, not a feature.

Why curated MCP sets win in production

Three reasons, none of them about model capability.

Security first. A fixed tool set is an allowlist. A synthesized tool has whatever side effects the model gave it, and the sandbox is the only thing between "compute a statistic" and "exfiltrate the dataset". The same week, on August 5, Anthropic shipped inference hooks that let an enterprise DLP server allow or deny each prompt and tool call before Claude generates. Note what that hook can see: the call, not the code the agent might write inside it. Dynamic tools slip under intent-level controls by construction, which is the same lesson as sandboxing coding agents.

Reproducibility second. A run that depends on tools evolved during that run cannot be replayed from the transcript alone. You need the synthesized code, its verification results, and the state that led there. That is a far heavier artifact than model, tool set and prompt versions.

Evaluation third. With a fixed tool set you can eval tool selection separately from the tools. With evolved tools every failure has two suspects, the reasoning and the code, and JudgeFlow-style attribution becomes mandatory.

A middle path that works today

Let the agent propose tools, but promote them through a human gate. Run the exploratory loop in a throwaway sandbox where synthesized code is allowed, log every tool the agent wrote and what it was verified against, and review weekly the ones that got reused. The survivors become real MCP tools with typed schemas and a code review. On a local stack you can wire this with the Ollama function calling pattern and a scratch container. The agent gets to evolve; production gets an allowlist.

The honest gap

I have not run either method, and the brief carries no benchmark numbers for either paper, so I cannot say how much it helps even on the authors' tasks. The brief flags both publication dates as unverified, so treat early August as approximate. What I am confident about is the security and reproducibility argument, because it does not depend on the papers' results at all.