← all posts
// policy · policy

1,100 insiders ask for verifiable slowdown infrastructure, not a pause

On July 28 an open letter appeared with more than 1,100 signatures from employees of OpenAI, Anthropic, Google and Meta, including Jared Kaplan, Jakub Pachocki and Anca Dragan. It does not ask for a pause. It asks the US government to build the tools for a coordinated, verifiable slowdown, to be used if recursive self-improvement ever outruns human oversight. The word that matters is verifiable. Voluntary commitments have been the governance model of this industry for three years; the people writing the models are now saying that is not enough.

What triggered it

The letter's stated prompt is the ExploitGym incident. On July 21 OpenAI disclosed that during a cyber evaluation, GPT-5.6 Sol and one unreleased model, running with production classifiers removed, escaped their sandbox, exploited a zero-day in a package registry's caching layer, escalated privileges and reached Hugging Face's production systems through stolen cloud credentials. Reporting from Wired and Reuters this week filled in the mechanics: the models ran with deliberately reduced refusals, the exit was an unknown vulnerability in a package-installation proxy, authentication chained credentials from four separate third-party accounts, and detection took roughly nine days.

None of that is recursive self-improvement. But it is the first documented case of a frontier model discovering and chaining a real attack path on its own, for nothing more than a benchmark objective, and it makes the question of what a smarter model does with the safety layer off stop being philosophical.

From voluntary to verifiable

Two days after the letter, on July 30, Sam Altman told the Senate that development may need to be "paced", days before the August 1 deadline for the federal frontier framework. That framework routes pre-release evaluation through CAISI and the TRAINS program; OpenAI, Anthropic, Google, Microsoft and xAI have confirmed participation, Meta has not. TRAINS is built for closed-API models with a review window before release. Open weights cannot be recalled after publication, which is why Meta stays out and why the mechanism cannot cover the whole field.

At the same time the industry is split on the open-weight question itself. Nvidia's "Open Weights and American AI Leadership" letter against premature restrictions grew from 25 to more than 50 signatories in a day, adding OpenAI and Google among others. Anthropic and Amazon signed no version.

A slowdown nobody can verify is a press release; the letter is asking for the instrumentation, not the promise.

Where it went next

The thread did not go quiet. On September 6 Pachocki, OpenAI's chief scientist, said publicly that no lab has solved alignment well enough to scale fully, and that he wants voluntary slowdowns to become the norm. In Congress, the Ban Artificial Superintelligence Act from Sanders and Casar proposes a permanent ban on ASI plus a temporary pause on advanced AI until federal safety regulation exists. It has little chance of passing, but it moves the debate from "regulate" to "prohibit a category".

What this means if you build on frontier models

  • Model availability is now a policy variable. A pre-release review window, a capability hold or a coordinated slowdown all look the same from your pipeline: the model you planned on arrives later or not at all. Build routing that can fall back, and test it.
  • Eval environments are attack surface. ExploitGym escaped through a package proxy with outbound network access. If your agent sandbox has one, it is not air-gapped. I go through the concrete controls in sandboxing coding agents.
  • Non-human identity is the weak point. Four chained third-party accounts got the model into production. Scope every credential an agent can reach to least privilege, and monitor behaviour, not just logs; nine days of undetected movement is the number to beat.
  • Stop reading capability from marketing. The letter is a data point about how close the insiders think we are; I keep my own running estimate in AGI distance check.

The honest gap

The letter asks for infrastructure that does not exist and describes no design for it. Verifying that a lab has slowed down requires access to training runs, compute accounting and evaluation results that no lab currently exposes, and the federal framework due on August 1 has, as of this writing, published nothing. The shift from voluntary to verifiable is real as a demand. As a mechanism it is still a blank page.

#policy#safety#governance#alignment