A million-token context window — enough to hold a small library — has gone from a headline feature to a baseline expectation. It's genuinely useful. It's also widely misunderstood.
What long context is good for
Real gains: feeding a model an entire codebase, a long contract, or a full research corpus without chunking; keeping a long agent session coherent; reducing the plumbing of splitting and stitching documents. When the relevant information genuinely spans a huge input, long context removes a whole class of engineering pain.
Where "just paste everything" breaks
But capacity is not comprehension. Two problems bite:
- Cost and latency. Processing a huge context on every request is expensive and slow. Retrieving the relevant slice is often cheaper and faster.
- Signal dilution. More tokens can mean more distraction. Models still lose track of information buried in the middle of a very long input, and irrelevant context can actively degrade an answer.
A bigger window is a bigger desk, not a better reader. Piling everything on the desk doesn't help you find the one page that matters.
Long context and retrieval are partners
The mature view is that long context and retrieval solve different halves of the same problem. Retrieval decides what the model should look at; long context decides how much it can hold once it's there. The strongest systems use both — retrieve the relevant material, then give the model room to reason over it.
For builders, the lesson is to treat context as a resource to spend deliberately, not a bucket to fill. Put the right things in the window, near where they'll be used, and don't assume that more tokens automatically means a better answer.
