GLM-5.1: An evaluation set worth keeping

GLM-5.1 is worth a slot in your evaluation harness. That's the claim. The catch: a slot in the harness isn't a slot in production, and no benchmark table tells you which one you're looking at.

Z.AI lists GLM-5.1, not GLM-5.2, as its top coding model: 200K context, up to 128K output, agentic coding, long-horizon work, thinking, streaming tool calls, OpenAI-compatible access, identifier glm-5.1. A 5.2 isn't in the docs; prices and aliases shift fast, check again before you configure anything.

Judge it by the job, not the leaderboard

Start from the wrong end and you get a leaderboard-shaped decision: pick the model, then invent a reason it fits. Reverse it: requests, deadlines, evidence, the tools involved, what a bad answer costs, then ask whether this model changes the picture enough to earn a route.

Decide what success looks like before the first prompt, in terms code can check: a schema-valid record whose fields match the source evidence, a patch that passes tests and review, a claim set with sources you can trace.

A fluent answer and a correct one are not the same test.

The method is unglamorous: collect real failures, strip anything sensitive, define the properties a good answer must have, score outputs blind, repeat after every change. Keep the whole request, not just the output: instructions, tool schemas, files, sampling or reasoning settings, output cap. Give one harness native tool calls, hand the other pasted terminal text, and you'll blame the wrong model.

Where long context and tool calls bite

The trap is building a test set easy enough to confirm the purchase and never sharp enough to catch a regression. Frontier output makes this worse: nearly everything it produces reads as plausible. Measure what code can verify, have a reviewer name the defect, and keep the bad transcripts, worth more than another folder of successes.

Long context bites hardest. A bigger window doesn't remove prefill time, quiet retrieval noise, or separate trusted instructions from fetched documents. Put stable content first so caching works, keep volatile metadata off the cache boundary, and send a structured checkpoint instead of full history when you can. Output deserves the same restraint: a short decision with evidence beats an essay nobody asked for.

Tool calling changes the risk, not the responsibility. A model picking the right function more often is nice; it doesn't validate arguments, check who's calling, cap side effects, or log what happened. Web pages, files, emails, and tool results are data, not instructions. A sharper model doesn't replace a narrower interface.

The lane it earns, not the lane it's given

The decision worth keeping is boring on purpose: promote a model only when it clears the task's gates and its failures have a safe way to get caught. Put that in configuration, not tribal memory.

Log model identity, phase timings, token counts by category, what each tool call did, and the evaluation result, without keeping sensitive prompt content by default. Keep a cheaper, faster lane for routine work and a clear escalation trigger. Give the strongest model tasks that need its capability; stop assuming a cheap tier is a bargain once retries and review eat the savings.

Then treat every model swap as an operational event: pin dated versions, watch deprecation notices, canary new aliases before they take your traffic, keep a trusted fallback. Re-run your own set whenever the model, prompt, tool contract, or reasoning default changes underneath you.

The rule I keep: no model touches production traffic until it has beaten my own failure set, on my harness, and I know what it hands off to when it can't.