Grok 4.5: An evaluation set worth keeping

What the vendor page actually tells you

The model ID is grok-4.5. That's the fact in this piece I'd stake money on, because previews get renamed, prices shift, and the exact wording on xAI's docs page will read differently by the time anyone acts on it. Check the current documentation again before you wire this into anything that matters. What it says today is unremarkable in a useful way: xAI lists Grok 4.5 as the active generation across chat and connected workflows, a frontier assistant aimed at chat, files, connected tools, multimodal input, and tasks that need current information. Fine. None of that tells you whether it beats the model already routing your extraction jobs, or your code review jobs, or whatever actually pays the bill here. A product description is not a decision. Decisions come from your own failures, not from xAI's marketing copy.

Define what winning looks like before you spend a token

Before sending a single prompt, decide what a correct answer has to contain for the job in front of you:

• extraction: a schema-valid record whose fields trace back to source evidence
• coding: a patch that passes tests and survives review
• research: a claim set with sources a person can actually check

Fluency doesn't count toward any of these. A confident wrong answer and a hesitant correct one score identically if tone is all you're reading.

The real test is duller than it sounds: pull real failures out of production, strip anything sensitive, write down what a correct output must contain, score outputs blind, and run the whole thing again the next time a model or a prompt changes. Keep the complete request when you do this, not a tidy summary of it, meaning system instructions, tool schemas, attached files, sampling and reasoning settings, and the output limit, because a model can look meaningfully better or worse purely on whether the harness hands it native tool calls or makes it chew on pasted terminal output. Hold the workflow itself constant across comparisons and let each provider use its own documented message format. That distinction, harness versus model, is where most head-to-head comparisons quietly go wrong.

Then log the run itself:

model id + dated API version
prompt and tool-schema revision
input / cached input / reasoning / output tokens
queue time, first token, total completion
pass / retry / abstain / human repair
provider error and fallback route taken

None of that is exotic. It's the difference between an opinion about grok-4.5 and a record you can defend six months from now.

The mistake that costs you later

The expensive failure isn't picking the wrong model. It's building a test set easy enough to confirm whatever you already wanted to buy, one that never catches a real regression. Frontier output makes this worse, not better, because almost every response reads as plausible on a skim. Fight that by measuring properties ordinary code can check, having a reviewer name the actual defect instead of a vibe, and keeping the failed examples instead of deleting them once you've made your point. A folder of failures is worth more than another page of clean transcripts. It's what tells you where to route around the model later.

Long context deserves the same suspicion, maybe more of it. A bigger window doesn't erase prefill time or quiet retrieval noise. It doesn't solve the problem of keeping trusted instructions separate from whatever untrusted document just got pasted in, either. If the API supports prompt caching, put your stable prefix first and keep anything volatile, timestamps, session IDs, away from the cache boundary, or you'll pay full price on every call without knowing why. Don't dump an entire conversation history in either, when a short structured checkpoint would do the same job for a fraction of the tokens. Output deserves the same discipline: a short decision plus its evidence is cheaper to generate and easier for a human to check than an essay nobody asked for.

Tool use changes the risk, not just the cost. A better model might pick the right function more often, but your application still has to validate every argument, confirm the caller is allowed to do this, cap the blast radius of anything with side effects, catch duplicate calls, and log what actually happened. Web pages, uploaded files, emails, and tool results are data the model reads, not instructions it should obey. However capable the model gets, that's no reason to remove the narrow interface standing between it and anything real.

Where this actually earns a place in the router

Promote grok-4.5, or anything else, only once it clears gates built for the specific job and its remaining failures fail safely. Write that rule into configuration. Don't leave it to whoever's excited about last week's release notes to decide by feel, and I say that having made exactly that mistake more than once. In production, log model identity, phase timings, token categories, tool outcomes, and evaluation results, and don't retain sensitive prompt content by default just because you technically can.

Keep a cheap, fast lane for the routine share of traffic and a defined trigger for escalating into the expensive tier. Send the strongest model the requests that actually benefit from its extra capability, not everything the gateway happens to touch, a version of the same dial I wrote about for routing tasks to the right model. A cheap model earns nothing if its retries and the review time it generates cost more than the inference it saved you.

Treat migration like the routine operation it is, the way I've argued for treating model swaps as normal operations instead of events: pin dated versions where stability actually matters, watch for deprecation notices instead of waiting for a broken call to tell you, canary any new alias before it touches all your traffic, and always keep a known-good fallback wired in. Re-run the local eval set every time the model, the prompt, the tool contract, or a reasoning default changes, because any one of those four can quietly move the numbers you built your trust on.

I'd rather run that whole loop and miss a week of whatever gain grok-4.5 has over the model already in production. Slower rollout, in exchange for never learning about a regression from a user instead of a test.