GDPx: the best agent scores 12%, and the verifier is the real product

The best AI agent on SemiAnalysis's new benchmark scores 12.0%, even with up to eight hours and three attempts per task. If you are about to hand a real workflow to an agent, the grading behind that number is the part to copy. GDPx launched at the start of November with 206 agentic tasks across 11 domains SemiAnalysis considers the most valuable for frontier AI, from chip design and quant trading to biology, law and alignment, with each model in its own harness. Claude Opus 5.5 tops the leaderboard. In a year when every launch post claims agents can replace whole teams, that is the most useful result I have seen.

How GDPx runs a task

Models run in their native harness where one exists: Claude Code for Opus 5.5, Codex for GPT-6 Astra, Antigravity for Gemini 4 Carbon, and OpenCode for everything else. Fourteen specialist data partners built the tasks, and Modal provides the sandboxes. The headline is pass@1 averaged over three attempts, with all eleven domains weighted equally.

The environments are big. The finance domain simulates a private equity deal for a fictional snack company with about 4,000 files, roughly 2,000 of them in the deal room. Under strict binary grading nearly every model scores zero there, so the leaderboard reports partial credit instead. Quant trading is my favourite design: the agent saves a strategy, it is simulated on hidden out-of-sample days, and a one-sided paired t-test on daily P&L against a reference strategy decides the score. A p-value under 0.01 earns full marks, no language model touches the grade, and the reference is scaled up to match any extra risk the agent takes, so a strategy cannot buy a better score by betting bigger. Of 111 graded runs, 70 underperformed the reference.

The cost column matters as much as the score. Gemini 4 Carbon gets within about a point of the leader at under a third of the price per task, and DeepSeek 4.1 Flash lands at almost half the leader's score for under a dollar.

Deterministic verifiers against rubrics

The comparison SemiAnalysis draws is with OpenAI's GDPval from September 2025. GDPval tasks are non-interactive, with all context in a prompt of around 500 words asking for a document of up to 15 pages, and grading is rubric-based. SemiAnalysis cites 71% agreement between human expert graders on GDPval's gold subset and says the automated grader did worse. When your judges agree seven times in ten, a two point gap between models is noise.

GDPx pushes toward deterministic checks wherever the domain allows it. The biology partner grades everything deterministically because it saw inconsistent judges and false positives with rubrics. Hardware design pairs deterministic verifiers with an LLM judge on expert rubrics. Legal still uses an LLM judge, but calibrated against expert labels and run five times per output. SemiAnalysis also claims its partner QA would flag nearly half the tasks in Terminal Bench 4.0 as broken, which I have not checked, but which matches my experience of public agent benchmarks.

A benchmark is only as honest as its grader, and a grader you can rerun and get the same answer from is worth more than any model on the leaderboard.

What the failures look like

The software engineering section is the part I would print out for my team. Picture the pull request: the agent adds the feature, its new tests pass, and an edge case the ticket asked for was never handled. Of the failed rollouts, 76% involved requirements tied to the main request, such as preserving existing behavior and handling edge cases. Of all failures, 24% were requirements the agent never attempted, 23% were misread intent and 17% were recognised but left unfinished. The green suite is part of the problem: the models tend to write tests that confirm their own code rather than the behavior that was asked for. I see that failure daily, and I covered the judge side of it in evals and LLM-as-judge.

The caveats

GDPx is not neutral ground either. Equal weighting of eleven domains is an editorial choice; weight software engineering higher and the ranking could move. Native harnesses mean you are comparing model plus harness, not models alone, and Claude Code versus OpenCode is not a controlled variable. The data partners are commercial vendors of exactly this kind of evaluation work. And 12% is an average that includes domains where everyone scores near zero, so it says little about any single workflow of yours. What it does refute is the claim that frontier agents already do eight hours of expert work unattended, which also squares with the agent ROI gap I keep seeing in real deployments.

What to copy from it

So before you build an agent for a workflow, take the one task you most want to hand off and write its verifier first. For a code change, that is a test suite written from the ticket, including the existing behavior and edge cases you would otherwise forget, not the tests the agent writes for itself. For a ledger, a reconciliation that must balance. For a trading strategy, a statistical test against a baseline scaled so that taking more risk cannot buy a better score. Where no deterministic pass exists, use a rubric judge, calibrate it against human labels and run it several times, as GDPx's legal domain does five times per output. Then run your agent against that verifier, per domain, with cost per task next to the score. If you cannot write the verifier, you are not ready to automate the task.