One sentence about re-running the simulation nearly triples engineering-agent success

A preprint on arXiv (2608.28147, August 2026) ran a controlled experiment that every team deploying engineering agents should read twice. Five Qwen models, eight synthetic engineering tasks in the DWSIM process simulator, two conditions: with and without one explicit instruction to re-run the simulation after any substantive change. With the instruction, agents re-verified in 94 of 120 runs versus 32 of 120 without. Cadence violations, meaning the agent changed something and did not check, dropped from 87 of 120 to 26 of 120. Bounded success, the actual task outcome, went from 35 of 120 to 95 of 120. One sentence, nearly three times the success rate, same models.

What the experiment measured

DWSIM is a chemical process simulator; the tasks are the kind of thing a process engineer does, with a ground-truth check available by running the simulation. That makes it a clean testbed: verification is cheap, deterministic and unambiguous, so the only variable is whether the agent chooses to do it.

The finding is that agents mostly do not, unless told. Without the instruction, models made substantive changes and moved on 87 times out of 120. They had the tool; they simply did not treat re-verification as part of the loop. With the instruction, the same models re-verified in 94 runs and their success rate nearly tripled. The capability was there; the policy was missing.

Why this is a policy result, not a model result

The instinct when an agent fails is to try a bigger model. This experiment says the cheaper lever is upstream. Across five different Qwen models the effect held: the cadence instruction moved the outcome more than the choice of model did. Reliability lived in the prompt and policy layer.

Your agent's reliability is mostly a property of what it is told to do after every change, not of how smart it is.

There is a corroborating data point from the same week. Anthropic reported on September 4 that Claude, mostly autonomously over about 11 days, produced a computer-verified Lean proof of Fermat's Last Theorem: around 13 million lines of Lean, 30,300 theorems proved, roughly 6 billion output tokens, orchestrated as multiple agents working a directed acyclic graph of sub-claims. The interesting choice is that every sub-result was gated by a machine verifier before the next step could build on it. That is the DWSIM instruction at industrial scale: no substantive change survives without a check.

How to encode verification cadence

The experiment used a single sentence. In production you want the same rule enforced in more than one place, because prompts drift and agents skip steps.

• Put the cadence in the system prompt or agent policy file in plain words: after any substantive change, run the test suite, the simulation, the type checker, whatever the ground truth is, before doing anything else. Name the tool.
• Version the policy. When you change that sentence, you change the agent's success rate. Treat the file like code, with a diff and a date; a small team can do this with the prompt versioning setup and nothing else.
• Enforce it outside the model. A post-tool hook that blocks the next edit until the verifier has run since the last one turns a suggestion into a rule. If you use a repo-level agent file, the AGENTS.md conventions are the natural home for it.
• Measure cadence violations as a metric. Count the runs where a change was followed by another change without a verification in between. The paper's 87 versus 26 is that metric; you can compute it from your own traces.
• Gate outputs, not just steps. An output validator at the end catches what the cadence rule missed and gives you the bounded-success number to track over time.

The honest gap

This is a preprint, on synthetic tasks, in one simulator, with one model family. Five Qwen models is not the frontier and DWSIM is not your codebase; a tool-use-tuned frontier model may already re-verify more often by default, which would shrink the gap. The 120-run denominators are small enough that a few runs move the percentages. What survives those caveats is the direction and the cost: the intervention was free, the effect was large, and it worked across every model tested. Add the sentence today and measure whether it holds on your tasks.