MegaFlow: splitting model, agent, and environment into services to run tens of thousands of agents

The MegaFlow paper landed on arXiv in early August (the exact date is worth verifying against the listing; I picked it up through the VoltAgent collection of 2026 agent papers). The claim is simple: a distributed orchestration system that separates agent execution into three independent services, Model, Agent, and Environment, and schedules tens of thousands of concurrent agent tasks across them. I have not benchmarked it and the summary I read gives no throughput numbers, so this is not a review. It is a look at the one design decision in it that every production agent runtime eventually has to make, usually after getting it wrong once.

The three services

Model is inference: tokens in, tokens out, batched and served. Agent is the loop: prompt assembly, tool selection, the decision of what to do next. Environment is where side effects happen: the shell, the browser, the filesystem, the API the agent is calling. MegaFlow's contribution, as described, is treating these as separately scaled, separately scheduled services rather than one process that does all three.

That sounds obvious written down. It is not how most agent code starts. The typical first version is a single process: it calls the model, parses the response, runs the tool in the same process, appends the result, and loops. It works at one agent. It works at ten. Somewhere between a hundred and a few thousand it becomes the most expensive component in the system, and not because of the model.

Why model serving and environment must not share a process

• They scale on different axes. Inference wants big batches on expensive accelerators, kept busy. Environments want many cheap, isolated sandboxes that sit idle most of the time waiting on an agent's decision. One process means sizing the expensive resource for the cheap one's concurrency.
• They fail differently. A sandbox that hangs on a network call, or a tool that eats memory, should kill that environment, not the inference worker serving a thousand other agents.
• They have different trust levels. The environment runs untrusted output. The model server holds keys and weights. The sandboxing rules for one are the opposite of the performance rules for the other.
• They have different lifetimes. An environment lives for one task. A model server lives for weeks. The agent loop in the middle is the only thing that should know about both.

If your agent loop can crash your model server, you have not built an agent system. You have built a very expensive single point of failure.

What the scheduler has to know

Once the three are split, the interesting engineering is the scheduler MegaFlow puts in the middle. At tens of thousands of concurrent tasks it has to answer: which agent step is ready to run, whether its environment is warm or needs provisioning, and whether the model tier has capacity at the latency the task tolerates. Practically, that means a queue per stage, backpressure between stages so a fast model does not flood slow environments, and a way to park tasks waiting on long tool calls without holding a model slot. It is the same shape as the continuous batching argument behind paged attention on the serving side: keep the expensive resource full, let the cheap resources wait.

Drawing the boundary in your own runtime

You do not need tens of thousands of agents to benefit. The moment you have more than one worker calling a shared model endpoint, you already have a Model service; make it explicit. The moment an agent runs a tool that can hang, you have an Environment; give it a process boundary and a timeout. What is left is the Agent service, which should be small, stateless between steps where possible, and the only thing holding the conversation. The notes in h100-vllm-team-serving cover the model side; the agent side is mostly discipline.

The honest gap

Everything above the checklist is architecture I agree with; everything about MegaFlow specifically is limited to what the paper says it does. I have no numbers on its scheduler throughput, no comparison against a naive runtime, and no idea how it handles agent-to-agent communication or failure recovery at scale, because the summary does not cover them. Read the paper before you cite it. Split your services either way.