The context window is bigger than the context you can use

When a model ships with a million-token context window, the marketing implication is that you can stop being careful. Throw the whole codebase in, dump every document, paste the entire conversation history, and let the model sort it out. No more fussing with retrieval, no more deciding what's relevant. It all fits.

It does all fit. That's true and it's not the point. The number on the box is how much the model can accept. It is not how much the model can use well, and the gap between those two things is where a lot of long-context applications quietly underperform.

Attention isn't flat

A model doesn't pay equal attention across a long context. It reasons best near the beginning and the end, and it gets hazier through the middle. This is the "lost in the middle" effect, and it's well documented: take a fact the model needs, bury it at the 60% mark of a very long prompt, and the model's ability to find and use it drops noticeably compared to putting the same fact near either edge. The information is technically there, sitting in the context, fully accepted by the API. The model just doesn't reliably reason over it.

So "it fits in the window" and "the model will use it" are different claims. You can hand a model a million tokens and have it functionally ignore the one paragraph that mattered because you happened to place it in the dead zone.

Retrieving a needle is not reasoning over a haystack

There's a benchmark people point to called needle-in-a-haystack, where you hide a single sentence in a giant context and ask the model to find it. Modern long-context models pass it, and that gets cited as proof the long context "works." But finding one planted sentence is a much easier task than reasoning over the whole document. Synthesizing across fifty places in a long context, holding all of it in mind at once, noticing that page 3 contradicts page 47, that's the thing that degrades as the context grows, and it's the thing you usually want. Passing the needle test tells you the model can retrieve from long context. It tells you very little about whether it can think over it.

Big prompts aren't free, in money or in time

Every token you put in the window is a token you pay for, on every turn, and a token the model has to process before it can respond. A million-token prompt is a million-token bill and a noticeably slower time-to-first-token. For a batch job that reads one enormous artifact once, fine. For an interactive loop that resends a bloated context every turn, you've signed up for a slow, expensive experience, and you've done it in exchange for headroom you're mostly not using well anyway. The KV cache for all that context also has to live in memory, which is its own cost that surprises people when a long-context request runs out of room the weights had no trouble fitting into.

The window is a budget, not a bucket

Here's the mental shift that fixes most of this. Stop treating the context window as a bucket you fill because you can, and start treating it as a budget you spend on the most relevant tokens, placed where the model actually looks. Curate even when you don't have to. Put the material that matters near the top or the bottom, not the murky center. If you can answer the question with three relevant chunks, send three, not the whole corpus, because the three dense, relevant tokens will be reasoned over better than the same three buried in a million tokens of context the model is skimming.

Here's what trips people up: a large context window does not retire retrieval. It changes when you reach for it. The window is the right tool when the problem genuinely is one big artifact you need to reason over end to end, an unfamiliar codebase or a sprawling log. It's the wrong tool when you're using "it all fits" as an excuse to skip the work of figuring out what's relevant, because the model still reasons better over a curated, well-placed context than over an undifferentiated dump, even when the dump fits.

The capacity is real and useful. Just don't confuse the size of the container with the quality of attention the model pays to what's inside it. The number on the box is a ceiling, not a promise.