Since 2017, one architecture — the transformer — has powered essentially every major AI advance. That's an extraordinarily long reign in a fast field. Naturally, researchers keep asking: what comes next?
Why look past the transformer
The transformer's dominance rests on attention, which is powerful but has a well-known flaw: its cost grows with the square of sequence length, making very long contexts expensive. That single weakness motivates most of the search for successors — architectures that could match transformer quality without the quadratic wall.
The contenders
- State-space models (like Mamba) process sequences in linear time with constant memory — cheap on long inputs. They've closed much of the quality gap but haven't fully surpassed transformers alone.
- Hybrids — mixing attention with SSM or linear-attention layers — are the pragmatic front-runner, keeping attention's quality where it matters and cheap layers elsewhere.
- Linear attention variants approximate attention at lower cost.
- More exotic ideas — new forms of recurrence, memory-augmented designs — appear regularly.
The successor to attention may not replace it, but absorb it — one primitive among several, used only where it earns its cost.
Why dethroning is hard
The transformer has a massive moat that has nothing to do with elegance: the entire ecosystem — hardware, software, training know-how, optimizations like FlashAttention — is built around it. A challenger must be not just better in theory but better enough to justify abandoning all that accumulated infrastructure. That bar is why, even with strong alternatives, transformers (increasingly in hybrid form) still dominate.
The likely future
The honest bet for the near term isn't a clean replacement but evolution: transformers augmented and hybridized, borrowing linear-time ideas to handle long context while keeping attention's strengths. "What comes after the transformer" may well still be called a transformer — just one that has quietly absorbed its rivals' best ideas.
