Mixture-of-Experts made computation conditional by choosing which parameters to use per token. Mixture-of-Depths applies the same idea to a different axis: how many layers a token passes through.
The idea
In a standard transformer, every token flows through every layer — the same amount of computation whether it's a trivial word or a pivotal one. That's wasteful; not all tokens are equally hard. Mixture-of-Depths lets a router decide, at each block, whether a token should be processed by that layer or skip it entirely. Easy tokens take a shortcut; hard tokens get the full depth.
Some tokens deserve deep thought. Most just need to be passed along. MoD stops spending the same on both.
Why it saves compute
By routing only a subset of tokens through each layer, the model does less total work for the same sequence — or, equivalently, can afford more capacity for the tokens that matter. It's conditional computation: spend where it counts, save where it doesn't. This composes with sparsity (MoE) — you can be sparse in both which experts fire and which layers run.
The broader theme
Mixture-of-Depths is one more move in the biggest efficiency trend of the era: making computation adaptive rather than uniform. Whether it's test-time compute (think longer on hard problems), MoE (use different experts per token), or MoD (use different depth per token), the pattern is the same — stop treating every input the same, and spend compute where it changes the answer.
