GPT-5.6: An evaluation set worth keeping

So no, three names on a tier chart don't settle anything. Sol, Terra, and Luna are OpenAI's new GPT-5.6 split, each with a 1.05M context window and a different price-performance slot: frontier reasoning, balanced production, cost-sensitive volume. That's the whole verified shape of it. Test gpt-5.6-sol, terra, and luna, and treat today's pricing and availability as provisional, it moves.

What counts as done, and who checks

Define the unit of success before the first prompt. Extraction needs a schema-valid record traced to its source; coding needs a patch that passes tests and review; research needs a claim set with sources you can follow. Fluency isn't the criterion for any of them. The method is dull and it works: collect real failures, redact them, define what a correct answer needs, score outputs blind, and repeat after every change, keeping the request constant, system instructions, tool schemas, files, sampling and reasoning settings, output cap. A model looks different when the harness hands it native tools instead of pasted terminal output.

The fields worth writing down

I log the same fields for every candidate: model identifier with its dated API version, the prompt and tool-schema revision behind the answer, the token split across input, cached input, reasoning, and output, the timing split across queueing, first token, and total completion, and an outcome bucket per task, pass, retry, abstain, human repair. Provider errors and the fallback route go in too. None of it is exciting: it's what tells you whether Terra got cheaper or just got lucky.

An easy test set is worse than no test set

The failure to watch for is an evaluation that confirms a decision already made and never blocks a regression; frontier output usually sounds right. Measure what code can verify, flag concrete defects, and keep the wrong answers, they shape routing and guardrails better than clean examples ever will. Long context deserves the same suspicion: a bigger window doesn't remove prefill time, retrieval noise, or the job of keeping trusted instructions separate from documents you didn't write. Cache stable content first, keep volatile metadata off that boundary, and keep answers short, a decision with evidence beats a tutorial. A sharper model isn't the one validating arguments, authorizing the caller, or logging what happened, your application is; what it reads stays data, never authority.

Make promotion a rule, not a mood

Promote a model once it clears task-specific gates and its failures have a safe fallback; put that threshold in configuration, not in whoever remembers which model is fashionable this month. Keep a cheap, fast lane for routine work and escalate when a request needs the frontier tier, since a cheap tier stops being cheap once retries and review eat the savings. Treat version migration like ordinary operations work: pin dated versions, watch deprecation notices, canary new aliases before full traffic, and keep a known-good fallback.

What I'd check next: pull last month's failures from whatever this replaces, redact them, and see how many Terra clears before touching a routing rule.