Reasoning models and test-time compute: when thinking is worth paying for

The biggest model-capability shift of the last two years is models that think before they answer, not a bigger model. Reasoning models generate an internal chain of deliberation, then respond, trading compute at inference time for quality. Knowing what that buys, what it costs, and when to dial it up or down is now core to using these models well.

How it works

A classic model does roughly one pass from prompt to answer. A reasoning model first produces a reasoning trace (often hidden), working the problem step by step, then writes the final answer. More thinking means more deliberation. This is test-time compute (a.k.a. inference scaling): instead of only scaling training, you spend more compute per query to get a better answer.

The dial has different names: "extended thinking," a thinking budget, effort levels (low → high → max), or adaptive thinking, where the model decides how much to think per request. The underlying tradeoff is identical: thinking tokens for quality.

The scaling law, and its ceiling

Quality improves as you spend more thinking tokens, up to a point. Then diminishing returns set in, and past that it can actively get worse. Yes, worse:

Reasoning doesn't follow "more is always better." Cranked to maximum on a simple task, a model can over-explore, second-guess a correct answer, and talk itself into a wrong one, and it bills you for the privilege. Effort is a dial to tune, not a slider to max.

The right setting is task-dependent, which is the whole skill.

When thinking pays

Spend the tokens where deliberation changes the answer:

• Hard, multi-step reasoning: math, complex debugging, planning, architecture decisions.
• Long-horizon agentic work, where one wrong early step cascades. Here, more thinking up front with a clear goal often reduces total cost because it prevents the expensive wrong turns the agent would otherwise grind through.
• Anywhere correctness ≫ cost and latency: the subtle bug, the migration that can't break.

When it doesn't

• Simple, lookup, classification, formatting, extraction: thinking is wasted tokens and latency. Run these at low or no thinking.
• Latency-sensitive interactive UX: reasoning shows up as a long pause before output. For a chat that needs to feel instant, that's a bug, not a feature.

The cost reality

Thinking tokens are billed (typically as output, the expensive side). A high-effort reasoning call can cost several times a direct answer. So effort is also a cost dial, and it belongs in the same toolbox as the other cost levers:

• Default to a sensible mid effort, raise it for genuinely hard tasks, drop it for routine ones.
• Route by difficulty. Low effort (or a cheap/local model) for the easy 80%; high-effort reasoning for the hard 20%. Reasoning effort is another tier in the cascade.
• Sweep it on your evals. The right effort per task type is an empirical question: measure quality-vs-cost on a golden set, don't guess.

The practical mental model

Treat reasoning effort as a per-task budget you set deliberately, the same way you'd choose a model. The mistake at both ends is treating it as a default: leaving everything on max (you overpay and risk overthinking) or on min (you underperform on the hard tasks where thinking was the point). Match the deliberation to the difficulty, measure it, and reasoning models become what they should be: a knob that buys correctness precisely where correctness is worth buying, and saves money everywhere else.