Context windows keep ballooning — millions of tokens now. It sounds like it should change everything: just paste in the whole codebase, the entire document set, the full history. The reality is more nuanced. A giant window genuinely helps some things and does nothing for others. Here's the honest breakdown.
What genuinely improves
More context is real capacity, and some workflows benefit a lot:
- Whole-document reasoning — analyzing a long report, book, or large codebase in one go, without chopping it up and losing cross-references.
- Fewer round-trips — you can include more relevant material at once instead of retrieving in pieces.
- Long conversations and sessions — more history fits, so the model "remembers" more of a session.
- Long-horizon agents — more room to hold plans, intermediate state, and tool outputs.
What doesn't change
A big window is capacity, not comprehension:
- Attention is uneven — models still attend best to the start and end and can miss things buried in the middle ("lost in the middle"). A bigger window doesn't fix this.
- Cost scales with tokens — filling a 2M window is expensive every call. More context isn't free.
- Relevance still matters — dumping everything in adds noise; the model can be distracted by irrelevant material. Curated, relevant context often beats a giant pile.
A huge window is a huge desk. It doesn't mean the model reads every page carefully — it means more pages fit. What you put where still matters.
The RAG question
Big windows didn't kill retrieval, as some predicted. Retrieval is still cheaper (send only what's relevant), often more accurate (less noise), and necessary when your data exceeds even a huge window. The best systems combine them: retrieve the relevant slice, and use the large window to hold enough of it, well-placed.
Why it matters
A 2M-token window is a powerful tool used well and a money pit used carelessly. It shines for whole-document reasoning and long agentic sessions; it doesn't replace good retrieval, careful context curation, or attention to where you place key information. Treat it as more room to be deliberate — not permission to stop being deliberate.
