Some breakthroughs are new ideas; others are new implementations of old ideas. FlashAttention is the second kind — it computes exactly the same attention as before, just far faster and with far less memory. That unglamorous win quietly enabled the long-context era.
The bottleneck it fixes
Attention's cost isn't only arithmetic — it's memory movement. A naive implementation writes a giant intermediate matrix (every token's relationship to every other token) to slow memory, then reads it back. For long sequences, shuffling that data dominates the time. The GPU spends more effort moving numbers around than computing them.
The math was never the whole problem. Moving data to and from memory was. FlashAttention fixes the plumbing, not the formula.
The trick
FlashAttention restructures the computation to avoid ever writing that huge matrix to slow memory. It processes attention in small tiles that fit in the GPU's fast on-chip memory, computing results incrementally. Same output, but a fraction of the memory traffic — so it runs much faster and uses far less memory, which matters most exactly when sequences are long.
Why it mattered
By making attention cheap in memory, FlashAttention made longer contexts and larger models practical to train and serve. It's now a default building block — the kind of infrastructure most people never think about but everything relies on. It's a reminder that in modern AI, hardware-aware systems engineering is as important as model design: a better implementation of a known idea can unlock as much as a new idea.
