You've probably heard of Mixture-of-Experts (MoE) — the trick that lets giant models like DeepSeek and GLM activate only a fraction of their parameters per token. But there's a quieter, equally clever idea making its way into frontier architectures: Mixture-of-Depths (MoD). Instead of choosing which experts to use, the model chooses how deep to go.
The Problem: Not Every Word Deserves the Same Effort
In a standard transformer, every token flows through every layer. Predicting the next word in "The capital of France is" gets the same 80 layers of computation as predicting the next word in a dense mathematical proof. Most of that compute is wasted on the easy tokens.
Think of it like a hospital triage system. A patient with a paper cut and a patient with chest pain shouldn't both get the full cardiology workup. But traditional transformers do exactly that — full workup for every token, every time.
How Mixture-of-Depths Works
MoD adds a simple decision point: at certain layers, each token is routed either through the layer's computation or around it (via a residual connection straight to the next block). A tiny router — a small learned network — makes the call per token.
The key ideas:
- Per-token routing: easy tokens ("the", "is", common phrases) skip layers; hard tokens (rare entities, ambiguous syntax, logic-heavy continuations) use them.
- A compute budget: the model enforces a fixed total budget of tokens per layer, which keeps inference predictable and batchable. It's not "sometimes faster, sometimes not" — it's a controlled allocation.
- End-to-end learning: the router isn't hand-programmed. During training, the model learns which tokens benefit from depth, discovering patterns researchers didn't design by hand.
Why It Matters: Speed You Can Feel
The original MoD research (from a Stanford team) showed that a model trained this way could match its dense counterpart while using dramatically less compute per token — or, at equal compute, score higher. Follow-up work on speculative-style decoding pushed it further: since skipping layers makes some tokens cheap, the model can even allocate more depth to the tokens that remain.
The practical results for users:
- Lower latency — responses start streaming sooner, because the average token takes fewer steps
- Lower cost — less compute per output token translates directly into cheaper inference on open markets like Qubax's model marketplace
- Same quality on hard problems — the depth is still there when the token needs it
MoD vs. MoE: Cousins, Not Competitors
| Mixture-of-Experts | Mixture-of-Depths | |
|---|---|---|
| What it saves | Parameters activated | Layers traversed |
| Router decides | Which experts process the token | Whether the token passes the layer |
| Savings type | Parameter efficiency | Computation-depth efficiency |
| Can combine? | Yes — some frontier models use both | Yes |
They're orthogonal: MoE spreads a big model across specialists; MoD paces the journey through the network. Expect both to show up together in the next generation of efficient flagships.
The Big Picture
AI compute is expensive, and most of it is spent uniformly on tokens that don't need it. Mixture-of-Depths is part of a broader shift — alongside speculative decoding, KV-cache optimization, and prompt caching — toward models that spend effort where it matters. That's also the philosophy behind Qubax: an open market where compute providers compete on price, so you only pay for the intelligence you actually use.
Curious how models like this are priced? Check out our weekly AI model price index or dive into the API docs to run one yourself.
FAQ
Is Mixture-of-Depths the same as early exit?
Related, but not identical. Early-exit models let a token leave the network entirely once it's "confident." MoD routes tokens around individual layers while still passing them through the rest of the network, and enforces a global compute budget that makes batching efficient.
Do routers ever make mistakes?
Yes — a token can skip a layer that would have helped. But because the router is trained end-to-end, it learns to err on the side of caution for hard tokens, and the residual path keeps the output stable even when it skips.
Which models use Mixture-of-Depths today?
It's appearing incrementally in efficiency-focused frontier architectures and research models. Check Qubax's live model catalog — pricing updates daily as providers compete, and efficiency gains like MoD are a big reason prices keep falling.
