Kimi K2.5: An evaluation set worth keeping

Somebody pastes a benchmark screenshot into the team channel, the score beats what's running in production, and by the end of the week the model swap has merged. Three weeks later someone flags an answer that sounds right and isn't, and nobody can say which prompt broke first, because nobody kept the losing examples. This has happened at more shops than will admit it, and it's the failure worth naming before you go near Kimi K2.5.

Moonshot's current platform docs list Kimi K2.5 as the multimodal Kimi release; there is no Kimi K2.7, and until there is, K2.7 doesn't belong in your code or your benchmark deck. The identifier to test is kimi-k2.5, built for long-context work, coding, agent loops, and document handling, wide enough that most teams find a use case in there. None of that is fixed: check Moonshot's docs again before you wire this in, since aliases and prices drift after this goes out.

What counts as done, not what sounds fluent

Before you touch the model, write down what a correct answer looks like for the job at hand, not the demo. Extraction means a schema-valid record whose fields match the source. Coding means a patch that passes tests and survives review. Research means claims you can trace back to a source. Fluency is cheap; K2.5 can sound like an expert and still be wrong in ways only the domain owner catches. Start from the requests you get, the deadlines on them, the evidence a reviewer needs, and what a mistake costs, then ask whether this model changes that math enough to justify switching anything.

Keep every answer that was wrong

The trap is a test set just rigorous-looking enough to always confirm the model you picked, because frontier output looks plausible on a skim. The fix is measuring what code can verify, having a reviewer name the actual defect, and keeping every wrong output. Wrong outputs are worth more than another folder of successes: they tell you where routing rules and guardrails need to go. Pull real failures from what you're running, strip anything sensitive, define the properties a correct answer would have, and score outputs blind. Do it again the moment the model or prompt changes; a passing run today says nothing about next Tuesday. Keep the full request, not a summary: system instructions, tool schemas, files, sampling and reasoning settings, the output cap. Skip that and harness differences will fool you: give one provider native tools, hand another pasted terminal output, and you're comparing two products on identical weights. Log the same fields for every model you compare:

• model ID and the dated API version, not just the family name
• prompt and tool-schema revision
• input, cached input, reasoning, and output token counts
• queue time, time to first token, total completion time
• task pass, retry, abstain, or human repair
• provider error and which fallback route it took

That list is boring on purpose. It's the only way to know if this got better, months later, instead of guessing.

Long context doesn't fix short judgment

A bigger context window gets sold as a solved problem, and it isn't. More room doesn't erase prefill time, doesn't quiet retrieval noise, and doesn't separate an instruction you trust from a document you don't; both sit in the same window and read the same to the model. Put your stable prefix first when the API supports caching, and keep volatile bits away from the cache boundary, or you pay for an unneeded re-encode. Skip full chat-history dumps when a structured checkpoint says the same thing in a tenth of the tokens. Ask for less back too: a decision and its evidence beat an unsolicited tutorial.

Tool use carries its own risk, and it doesn't shrink because the model got better at picking which function to call. Your application still validates every argument, confirms the caller is allowed, caps what a call can do, catches duplicates, and writes down what happened. A web page, a file, an email thread, a tool's return value: none of that is an instruction, whatever it claims to be. A sharper model doesn't replace a narrow interface that only allows what you meant to allow.

Pin it, gate it, log it, move on

The rule that holds up: a model gets promoted only when it clears the gates for the specific task, and whatever it still gets wrong has a safe way to fail. Put that in configuration, not in team memory, because whoever's on call in six months reaches for whatever's fashionable that week, not what passed your eval set. Log model identity, timing per phase, token counts by category, what the tools did, and the evaluation result, without hanging on to sensitive prompt content by default.

Keep a cheap, fast lane for routine work and a clear trigger to escalate. The strongest tier of model, and how hard you dial it up per task, goes to work that benefits from it, not every request the gateway passes. A cheap model needing constant retries and manual review isn't cheap; run the math on review time before trusting the invoice. Treat every version bump as a real event: migrations rarely go the way teams expect. Pin dated versions where stability matters, watch deprecation notices, canary new aliases before full rollout, keep a known-good fallback, and re-run your local eval set whenever the model, prompt, tool contract, or reasoning default changes.

None of that resolves the thing that still nags me. I said keep the wrong outputs because they're worth more than the right ones, and I also said don't retain sensitive prompt content by default. Real failures are usually wrong precisely because of what's in them, and the redaction that keeps you compliant is often the same redaction that erases the reason the thing failed. I don't have a tidy fix for that, and I'd be careful of anyone who tells you they do.