Research Preference Models: rank the experiment before you burn the GPU-day

Meta FAIR, Oxford, and UCL published AI Research Preference Models on arXiv (2608.13940; MarkTechPost covered it on September 6). The idea is narrow and, I think, correct. Instead of asking an LLM to predict what score a proposed experiment will get, which LLMs are bad at, ask it to rank a set of candidate experiments that have not been run and pick the one to execute. The output is a preference, not a number. That single change is why the thing works, and it is also why you can reproduce it on a workstation.

What the paper does

An RPM takes a set of candidate experiments, each described by a plan, code, and the history of what has been tried, and produces a ranking. There are two variants. The inference-only version reasons over the plans, code, and history and ranks from that alone. The agentic version is allowed to run small pilot experiments before ranking. The backbone is a frozen pretrained LLM with no fine-tuning, and the open-weight model used is Qwen3.6-27B. The scaffold, AIRA-dojo, and the benchmark, AIRS-Bench, are both open source.

The motivation is the cost of evaluation on frontier tasks: running every candidate burns days of GPU time. The RPM moves the expensive step from run everything to rank, then run one.

Why ranking beats prediction

Ask a model what accuracy a training run will hit and it will give you a confident, wrong number. Ask it which of two runs is more likely to be better and it is on much firmer ground, because the comparison can lean on structure: which change is bigger, which has been tried before, which fixes a known failure. Preference is also robust to the calibration problems that wreck absolute predictions. It is the same reason pairwise judgments beat scoring in LLM-as-judge setups, applied to the research loop itself.

You do not need a model that knows the score. You need a model that knows which experiment to run next, and that is a much easier question.

The number that makes this urgent

The same week, OpenAI announced it had hit its automated research intern goal. Its research organization now logs 3.1 agent-workdays per human workday; the median researcher spends over $600 a day on inference, and the 90th percentile runs past $7,000 a day in tokens. OpenAI itself warns that 3.1x activity is not 3.1x output: agent runtime is parallel, redundant, often unsuccessful, and heavily steered. That is exactly the failure an RPM targets. When agents generate candidate experiments faster than you can afford to run them, a ranking gate is the difference between a research budget and a token bonfire.

Reproducing it locally

The reason I am writing this up rather than filing it is that the whole stack is open. A frozen Qwen3.6-27B is a model I already run; the ThinkingCap notes cover the serving setup. Here is what I would do:

• Run AIRS-Bench with the inference-only variant first. No pilots, no GPU spend beyond the ranking calls. Measure how often the top-ranked candidate is actually the best when you do run them all.
• Cost the ranking step honestly. Tokens per ranking call times candidates per round. On a 27B model served locally this is cheap; on a frontier API it is not free. Use the quality-adjusted token speed framing: what you care about is correct rankings per dollar, not tokens per second.
• Add the agentic variant only where pilots are cheap. A five-minute pilot that improves ranking accuracy is worth it. A two-hour pilot is just a smaller version of the original problem.
• Gate your own pipeline. Any place where an agent proposes N experiments and you run all N is a place to insert the ranker and run the top one or two.

The honest gap

I have not run AIRS-Bench yet, so I cannot tell you how much better than random the ranking is on your kind of experiment, and the paper's results are on its own benchmark. Ranking quality will depend on how well the candidates are described; vague plans produce vague preferences. And the OpenAI cost numbers are self-reported by a lab with reasons to publicize both the adoption and the warning. Treat the RPM as a promising gate, not a proven one, and measure the hit rate before you trust it with a GPU budget.