A few years ago, Mixture-of-Experts was a specialist technique with a reputation for being finicky to train. Today it's the default: nearly every serious open-weight release is sparse. The switch wasn't fashion — it was economics.
The core bargain
A dense transformer runs every parameter for every token. An MoE layer replaces that with many expert sub-networks and a small router that sends each token to just a few. You get the capacity of a huge model at the compute of a small one. When you're paying for every token you serve, that bargain is hard to refuse.
More capacity, similar compute. That one sentence explains most of 2026's architecture choices.
Why it took over now
The problems that made MoE hard — routers collapsing onto a few favorite experts, unstable early training, heavy cross-device communication — turned out to be tractable at scale. Load-balancing losses keep expert usage even. Training recipes stabilized. And once a handful of labs proved sparse models could match dense frontier quality, everyone followed, because the serving cost advantage compounds at production volume.
What's still hard
Sparsity moved the frontier of difficulty rather than removing it. The open questions are now about routing quality — does each token reach the expert that actually helps it? — and about the systems engineering of serving experts spread across many devices without communication swamping the gains.
For practitioners, the takeaway is simple: if you're comparing models, "total parameters" and "cost to run" no longer move together. Sparsity broke that link, and it's why the biggest open models are also, increasingly, the ones you can afford to run.
