Context windows grew to a million tokens. The headlines followed the length. The economics, though, are decided somewhere less glamorous: the KV cache, the memory where a model stores what it has already read.
Where the cost actually goes
When a model processes a prompt, it stores a key and value vector for every token at every layer. That store grows linearly with context length and, at long contexts, it — not the model weights — is what fills the GPU. Doubling the context doubles the cache. This is why serving a 200K- or 1M-token context is disproportionately expensive: you're paying for memory, not just math.
Context length is a marketing number. KV-cache size is the bill.
The fixes
A whole line of engineering exists to tame the cache:
- Grouped-query attention shares keys and values across query heads, shrinking the cache several-fold.
- Sliding windows cap how far back attention looks.
- Quantized caches store keys and values in fewer bits.
- Hybrid architectures avoid a per-token cache entirely for some layers, using recurrent or state-space designs.
Why it keeps mattering
Every advance in context length reopens this problem at a larger scale. The million-token window is impressive precisely because the cache math is brutal — and the systems work to make it affordable is where much of the real progress happens. Long context isn't only an accuracy achievement; it's a memory-systems achievement, and the KV cache is where it's won or lost.
