AT&T cut AI costs 56% for a 2% quality loss: the cheapest model that reliably does the job

The Information reported this week that AT&T has cut the cost of a portion of its coding and AI workloads by up to 56% while giving up roughly 2% in quality. The method is not a new model. It is a router. Simpler tasks, code and document summaries in particular, go to open models like Llama and Gemma; frontier models from Anthropic and OpenAI get only the hard work. The scale makes this more than an anecdote: AT&T processes about 45 billion tokens a day, roughly 40% of requests already land on open models, and the stated target is 60 to 70%. Goldman Sachs, via Jim Covello, has turned the same observation into an investment thesis: the model optimization layer is the enterprise AI bottleneck.

The question changed

For two years the enterprise question was which model is best. The AT&T numbers answer a different question: which is the cheapest model that reliably does this task, and how do we know. Asked that way, models become interchangeable compute for routine work, and the value moves to two places: the orchestration that decides, and the measurement that proves the decision was right. The same week, DeepMind reported the Gemma family passing one billion downloads with over 100,000 community variants. The workhorse tier is not hypothetical. It is the most downloaded thing in the stack.

Stop asking which model is best. Ask which is the cheapest one that reliably does this task, and how you would know if it stopped.

A reference router

Here is the shape I build when a client wants their own version of this. It is four components, and the fourth is the one everyone skips.

• Difficulty classification. Before a request reaches a model, something decides its tier. For code and document summarization, cheap heuristics get far: input length, whether the task is extractive or generative, whether it touches more than one file, whether the prompt asks for reasoning. Where heuristics fail, a small classifier model, itself on the cheap tier, makes the call. Log the decision.
• Fallback. The cheap tier will fail some requests. Define failure explicitly (malformed output, validation error, low confidence, user retry) and escalate to the next tier automatically. A router without fallback is a cost cut that silently becomes a quality cut.
• Per-task evaluation. AT&T's 2% is a per-task number, not a global benchmark. You need a held-out set for each task class, scored the way that task is actually judged, and you need to re-run it whenever a model or prompt changes. This is the eval harness, and it is what makes the 56% defensible in front of a CFO.
• Observability. Per request: tier chosen, model used, tokens in and out, latency, cost, eval outcome if sampled. Per task: cost per successful outcome, escalation rate, quality drift over time. If you cannot produce the 56% and the 2% from your own logs, you do not have a router. You have a config file.

Where to start

Do not route everything. Pick the two highest-volume task classes, which for most engineering organizations are summaries and boilerplate generation, and route only those. Keep the frontier tier as the default for anything unclassified. The OpenRouter cost routing piece covers the plumbing if you want a hosted gateway; frontier-model-router-2026 covers tier design when you want to own the decision. Measure the escalation rate weekly. If it climbs, your classifier is wrong; if it drops to zero, your eval is too easy.

The honest gap

Every number here comes from The Information's account of AT&T's internal figures, and I have not seen the eval that produced the 2%. Quality loss on summarization is easy to underestimate when the metric is a similarity score and hard to see until a user acts on a wrong summary. The 56% is also "up to," on a portion of workloads, which is a ceiling, not an average. Treat the AT&T story as proof that the architecture works at scale, not as a promise about your numbers. Your numbers come from your eval, which is the whole point.