Same model, 5x the score: ARC-AGI-3 and the harness that matters more than the weights

On July 30 OpenAI published a GPT-5.6 Sol result of 38.3% on ARC-AGI-3, above Claude Opus 5 at 30.2%. The same post contains the number that actually matters: run through the official ARC harness, where reasoning is discarded after every action, the same model scores 7.8%. The 38.3% came only through OpenAI's own Responses API with two settings enabled. Retained Reasoning keeps the chain-of-thought between steps. Compaction summarizes old context instead of truncating it. Same weights, two runtime configurations, roughly a five-fold difference in score.

What was actually measured

ARC-AGI-3 is interactive: the model takes actions, sees results, takes more actions. That makes it an agent-loop benchmark as much as a model benchmark, and the loop's memory policy becomes part of the system under test. The official harness resets reasoning after each action. OpenAI's API path does not; it carries intermediate reasoning forward and compacts the transcript as it grows. François Chollet of the ARC Prize acknowledged the parity problem directly: general-purpose API settings anyone can turn on are fair, benchmark-specific harnesses are not. Retained Reasoning and Compaction are general-purpose settings, which is why the 38.3% is being reported at all rather than dismissed.

I care less about who won than about what the gap says. If the harness moves a frontier model from 7.8% to 38.3%, then on interactive tasks the harness is worth more than the difference between any two frontier models on the leaderboard. That is a strong claim, and this is the cleanest public evidence for it so far.

On agent tasks the loop is part of the model, and a benchmark that reports the score without the loop is reporting half a number.

The two levers are portable

Neither setting is exotic. Both map onto decisions you already make in a production agent loop, usually badly.

Reasoning retention is the question of what you feed back after a tool call. Most loops feed back the tool result and the prior messages and throw away whatever the model was thinking when it chose the action. The model then re-derives its plan from scratch every step, which is where drift and repeated mistakes come from. Keeping the reasoning costs tokens and buys consistency across steps.

Compaction is the question of what you do when the transcript gets long. Truncating from the front drops the original task statement first, which is the worst possible thing to drop. Summarizing older turns into a compact state block keeps the goal and the important discoveries while shedding the noise. It is the same idea as the KV cache eviction policy choice at the serving layer, applied one level up at the transcript.

How to report an agent benchmark

If you publish agent evals internally or to clients, this episode gives you the minimum disclosure list. I would refuse a number without it.

• Harness: which loop ran the model, and whether it is the benchmark's official one.
• Memory policy: is reasoning retained between steps, and what happens to old context: truncate, summarize, or nothing.
• Cost per episode: tokens in and out, and wall-clock time. Retention and compaction both change the bill.
• Settings availability: are the settings generally available to any API user, or specific to this run.
• Model ID and date: exact snapshot, because the name GPT-5.6 Sol will not mean the same thing in three months.

For your own loops the experiment is cheap: take one multi-step task you already measure, run it with reasoning discarded after each step and with reasoning retained, then add compaction at a fixed transcript size and run it again. Three configurations, one task, and you will know which of your model-quality problems are really loop-design problems. For how far these scores are from anything I would call general, the AGI distance check is where I keep that argument, and /benchmark tracks the models themselves.

The honest gap

Every number here comes from OpenAI's post and The Decoder's coverage of it. There is no independent replication of the 38.3% yet, no published token cost for the retained-reasoning run, and Opus 5's 30.2% was presumably measured under a different configuration, which makes the head-to-head weaker than the headline. What I trust is the 7.8% to 38.3% gap within one model, because that is the comparison OpenAI had the least incentive to publish.