Context windows have exploded — from a few thousand tokens to millions. Which raises a natural question: if more context is better, why is there a limit at all? Why not infinite? The answer explains a lot about how models actually work.
The quadratic problem
The core constraint is attention. In a standard transformer, every token attends to every other token — so the computation grows with the square of the sequence length. Double the context, quadruple the attention cost. At very long lengths, this becomes prohibitively expensive in both compute and time. Context length isn't free; it's one of the most expensive things you can ask a model to do.
The memory problem
There's a second wall: the KV cache. To process a long context, the model stores a key and value for every token at every layer. This memory grows linearly with length and, at long contexts, dominates the GPU — often before the compute limit even bites. You run out of memory to hold the context.
A bigger context window isn't just a setting to increase. It's a bill paid in compute (quadratic) and memory (linear), and both come due fast.
Why the limits are moving
Engineering keeps pushing the walls back: efficient attention (like FlashAttention) tames the compute, KV-cache quantization and hybrid architectures tame the memory, and clever position methods (RoPE extensions) let models generalize to longer inputs. That's how million-token windows became feasible.
The subtler limit
Even when a model can take a huge context, using it well is another matter. Models can lose track of information buried in the middle of very long inputs, and irrelevant context dilutes the signal. So the practical limit isn't only "how much can it hold?" but "how much can it actually use effectively?" — which is why retrieval, which feeds the model only what's relevant, remains valuable even in the long-context era.
