Latency is a feature: architecting AI apps that feel fast

Put two AI products side by side running the exact same model, and one will feel snappy while the other feels broken. The difference is how they handle the wait, not the model itself. One starts showing you words almost immediately and you read along as it writes; the other sits on a blank screen for four seconds, then dumps a finished paragraph. The second one is often technically faster end to end. It feels slower, and feeling is what ships.

This is the thing people get backwards about AI latency. They optimize the model's tokens-per-second, a number the user never directly experiences, and neglect the two things the user feels in their bones: how long before something happens, and whether the thing seems alive while it works.

Time to first token is the number

The metric that actually governs perceived speed is time to first token, not total response time. A response that begins streaming in three hundred milliseconds and finishes in eight seconds feels faster than one that appears complete after four seconds of nothing, because the first one started immediately and the user spent those eight seconds reading rather than waiting. Once tokens are flowing, the user's reading speed becomes the bottleneck, not the model's generation speed, and the wait turns productive.

That makes streaming the foundational architectural choice, not a nice-to-have toggle you add at the end. Stream tokens as they generate, over SSE or websockets, and the long total time stops mattering nearly as much because it's hidden behind text the user is already consuming. Build the same feature without streaming and you've chosen to make every response feel like its worst-case total latency.

Know where the wait actually comes from

Before optimizing, find your bottleneck, because the fixes are different for each. A large prompt makes the first token slow, because the model has to process all that context before it can generate anything, which is one more reason a bloated context window costs you beyond the token bill. A reasoning model at high effort spends real time thinking before it produces a visible token, so the pause is the deliberation, not the network. Each tool call in an agent loop is a round trip that adds its own gap. Then there's network and queue time. Optimizing prompt size when your latency is actually reasoning time, or vice versa, is wasted effort.

The reasoning case deserves special care, because it's a trap. A model that thinks for ten seconds before its first visible token is working correctly and feels completely broken in an interactive interface. You have two ways out: stream the thinking so the user sees the model is alive and working, or simply don't use high reasoning effort on latency-sensitive paths and route the deliberation-heavy work somewhere the user isn't staring at a spinner.

The same goes for tool calls. An agent that goes silent for five seconds while it runs a search feels stuck, even though it's doing exactly what you asked. Stream the intermediate steps, "searching the codebase," "reading config.py," so the gap fills with visible progress instead of dead air. Nothing about the actual work sped up, only the feeling of it.

Design the path to its requirement

There's an unavoidable triangle between cost, quality, and latency, and you can't max all three on the same request. The architectural move is to stop pretending you have one kind of request. An interactive, user-facing path should trade some quality for speed, a smaller or faster model, lower reasoning effort, a tight prompt, a warm cache so the first token comes fast. A background or batch path can trade latency for cost and quality, because no one is watching it stream. Route each request down the path that matches what it actually needs, which is the same routing logic you'd use for cost, applied to the latency dimension.

In an interactive product, latency is a feature you design for from the start, not a performance metric you patch on once the features work, because the model's raw throughput is invisible to the user and the whole experience is the experience of waiting. Make the wait short by getting to the first token fast, or make it productive by streaming, or both, and a perfectly ordinary model feels responsive. Skip it, and a frontier model feels like it crashed.