Chapter 13 · Section 13.6 · 45 min

When MoE is worth it

The pitch for mixture of experts fits in one line. DeepSeek-V3 trains with about one eighteenth of the FLOPs a dense model of its size would need. If that were the whole story, nobody would train a dense model again. Yet dense models are still trained, by careful labs, for good reasons. Where does the free lunch end?

It ends in five places, and this section visits each with the evidence. MoE training is less stable. MoE models are harder to fine-tune. They need memory for every expert while computing with only a few. Their efficiency depends on batch size in ways a dense model's does not. And the "experts" are not experts in the sense the name suggests, which matters for how you reason about them. We end with a decision table for when an MoE is worth it.

Training stability

Large sparse models were notorious for loss spikes and divergence. Dense models spike too, for the reasons of §8.8, but MoE adds reasons that are specific to routing. Router logits grow and bfloat16 rounds away their differences (§13.3). Exponentials in the router softmax amplify roundoff. And routing is discontinuous. A tiny change in a token's vector can flip its top-k set, and the layer's output then jumps, because a different expert, with different weights, suddenly processes the token. A dense FFN changes smoothly with its input. An MoE layer is smooth only between flips.

Python · cell 1

When the token's second expert changes from 7 to 0, the output jumps about fifty times further than a dense layer moves over the same small step. The size of the jump is the gate at the flip times the difference between the two experts' outputs, and the two experts have unrelated weights. At the flip the departing and arriving experts have tied logits, so their gate is the same, and it is small only if both are far behind the first expert. With thousands of tokens per batch, some are always at a boundary, and the router keeps moving them back and forth, which shows up as noise in the loss.

ST-MoE (Zoph et al. 2022) is the systematic study. With a baseline in which 4 of 6 runs stayed stable, they compared fixes on the same model, measuring quality as negative log perplexity (higher is better, baseline −1.755).

intervention	stable runs	quality	verdict
none (baseline)	4 of 6	−1.755	unstable
remove multiplicative GEGLU	3 of 3	−1.849	stable, worse
remove RMSNorm scale parameters	3 of 3	−2.020	stable, much worse
input jitter ( $10^{-2}$ )	3 of 3	−1.777	stable, slightly worse
dropout 0.1	3 of 3	−1.822	stable, worse
update clipping 0.1	3 of 3	−4.206	stable, ruined
router z-loss	3 of 3	−1.741	stable, slightly better

The pattern is that most stabilizers work by making the model less expressive, and the z-loss is the exception. Switch Transformers adds two more. Computing the router in float32 while the rest of the model runs in bfloat16 rescued a run that diverged in pure bfloat16 (quality −3.780) and matched float32 quality (−1.716 against −1.718) at bfloat16 speed. And initializing weights at one tenth the usual scale cut the spread across seeds from a standard deviation of 0.68 to 0.01.

Routing also settles early. OLMoE (Muennighoff et al. 2024) measured router saturation, the fraction of a token's top-k experts at an intermediate checkpoint that match those at the end of training. Up to about 60% of top-8 assignments already matched after 1% of pretraining, and up to about 80% after 40%, with later layers settling earlier than the first. So most of what the router learns, it learns early, and late changes in routing are a sign that something is moving.

Core

Router saturation between checkpoints

Implement router_saturation(R_mid, R_final, k) for router logits of the same tokens at two checkpoints, both of shape (T, E). For each token, take its top-k expert set at each checkpoint, count how many experts the sets share, and return the total shared over $kT$ , the fraction of assignments that already match the final ones. Order within a set does not matter.

import numpy as np

def router_saturation(R_mid, R_final, k):
    a = np.argsort(-R_mid, 1)[:, :k]
    b = np.argsort(-R_final, 1)[:, :k]
    return float((a == b).mean())          # placeholder: compares positions, so order matters

Hidden tests, hints, and AI help, from $1.99 a month.

Fine-tuning

Sparse models fine-tune differently from dense ones, and the differences are documented.

They overfit small tasks faster. With hundreds of times more parameters per FLOP, an MoE can memorize a small dataset quickly. ST-MoE found its sparse model reached 100% training accuracy faster than the FLOP-matched dense model and lagged it on held-out data for the tiny CB task (about 250 training examples), while beating it on the much larger ReCoRD. Switch Transformers found that raising dropout inside the experts to 0.4, while keeping 0.1 elsewhere, beat uniform dropout.

They want different hyperparameters. ST-MoE found that sparse models fine-tune best with smaller batches and higher learning rates, the opposite of the dense models in the same study, and that reusing dense settings can hide the sparse model's pretraining advantage entirely.

You can freeze the experts. Updating only the non-expert parameters (attention, norms, embeddings) worked about as well as updating everything, even though the experts hold about 80% of the parameters, and updating only the expert parameters was significantly worse. This makes parameter-efficient fine-tuning of MoE models more natural than it looks. Leave the experts alone and adapt the shared parts (§14.6 covers the methods).

Instruction tuning changes the picture. Shen et al. (2023) found that MoE models fine-tuned directly on downstream tasks underperformed dense models of the same compute, but after instruction tuning they did better, and they benefited more from instruction tuning than dense models did. Their FLAN-MoE-32B beat FLAN-PaLM-62B on four benchmarks with about a third of the FLOPs. The practical reading is that a broad, multi-task post-training stage (§14.2) suits MoE models better than narrow single-task fine-tuning.

Reinforcement learning adds routing noise. Policy-gradient objectives compare the probability of a sampled token under the current policy with its probability under the policy that generated it, an importance ratio (§15.6, §18.4). In an MoE model a gradient step also changes the routing, so the same token can be computed by different experts before and after the update. Zheng et al. (2025) report that for the 48-layer Qwen3-30B-A3B, about 10% of the experts a rollout activated changed after each RL update, which made token-level ratios fluctuate sharply. One fix, routing replay, records the experts used when the rollout was generated and reuses them when computing the ratio. Their GSPO method instead computes the ratio over whole sequences, which they found stable without replay.

Routing drifts under new data. A fine-tuning set from one domain can push the router to overload the few experts it prefers. Whether to keep the balancing mechanism during fine-tuning is a judgment call. In OLMoE's ablation, instruction tuning and preference tuning both scored slightly higher without the load-balancing loss (54.0 against 52.8 after SFT), though the paper's appendix describes keeping it, so the evidence is thin. Capacity limits, if any, still apply.

Memory versus compute at inference

An MoE model's FLOPs per token follow its active parameters, but its memory follows its total. Mixtral computes like a 13B model and occupies 93 GB in bfloat16, against 26 GB for a dense 13B and about 140 GB for Llama 2 70B. DeepSeek-V3 computes like a 37B model and occupies about 1.3 TB in bfloat16 (the released weights are stored in FP8, about half of that).

How fast each decoding step runs depends on how many bytes of weights it reads, because decoding is memory-bound (§6.5). Here MoE has a surprise in store. Take a decoding batch of $B$ tokens, one per sequence, each choosing $k$ of $E$ experts in a layer. If tokens route independently and uniformly, one token misses a given expert with probability $1 - k/E$ , all $B$ miss it with probability $(1-k/E)^B$ , and by linearity of expectation the number of distinct experts the step must read is

\mathbb{E}[U] = E\Big(1 - \big(1 - \tfrac{k}{E}\big)^{B}\Big).

In plain words, a single token touches $k$ experts, and as the batch grows the touched set fills up toward all $E$ , quickly when $k/E$ is large and slowly when it is small.

Three regimes follow. At $B = 1$ , $\mathbb{E}[U] = k$ , and an MoE reads only its active weights, so if everything fits in memory it decodes as fast as a dense model of its active size. At moderate batch sizes $U$ approaches $E$ while each touched expert serves only $Bk/U$ tokens. The step reads nearly all the weights, like a dense model of the total size, and does the arithmetic of the active size, the worst of both. Mixtral reaches 7.9 of its 8 experts by $B = 16$ , with about 4 tokens per expert. Only at large batches, when each expert gets hundreds of tokens, does the arithmetic catch up with the bytes (§13.4). For DeepSeek-V3, with $k/E = 1/32$ , the touched set fills slowly (about 102 experts at $B = 16$ , 222 at $B = 64$ ), but tokens per expert stay tiny until the batch is in the thousands.

Python · cell 2

Real routing is not uniform, and the Mixtral authors measured something that helps. Consecutive tokens tend to reuse experts. At layer 15, the first-choice expert of a token was the same as the previous token's 23.6 to 28.4% of the time, depending on the dataset, against 12.5% for random routing. That temporal locality is what makes expert caching and offloading on small hardware workable, and it also means that a real batch touches somewhat fewer experts than the uniform formula predicts.

Interview

Experts touched by a decoding step

Implement decode_expert_bytes(E, k, B, d, d_e, n_layers, bytes_per_param) for a model whose MoE layers each have $E$ SwiGLU experts of width $d_e$ and top- $k$ routing. Assuming uniform, independent routing, return (U, bytes_per_token), where U is the expected number of distinct experts one MoE layer reads in a decoding step with $B$ tokens, and bytes_per_token is the expected expert-weight bytes read per generated token across all n_layers MoE layers, $n_{\text{layers}}\cdot U\cdot 3\,d\,d_e\cdot s / B$ . Return floats.

def decode_expert_bytes(E, k, B, d, d_e, n_layers, bytes_per_param):
    U = min(E, k * B)                                    # placeholder: assumes no two tokens share an expert
    return float(U), float(n_layers * U * 3 * d * d_e * bytes_per_param / B)

Hidden tests, hints, and AI help, from $1.99 a month.

Batch size, in training and serving

The same arithmetic runs through training. Shazeer et al. (2017) named the shrinking batch problem. With a batch of $B$ examples, each expert sees only about $kB/E$ of them, so the more experts you add, the smaller and less efficient each expert's matrix product becomes. Their fix, combining data parallelism across $n$ devices with a single copy of each expert, gives each expert about $kBn/E$ examples, and it is the reason MoE training uses expert parallelism with large global batches (§13.4).

Batch size also sets how often tokens are dropped. A capacity buffer of $\text{CF}$ times the fair share overflows from random fluctuations alone, and the fluctuations shrink relative to the mean as the routing group grows (§13.2 computed 13% drops for 64 tokens over 8 experts at $\text{CF} = 1$ , and 1.6% for 4,096).

And there is a subtle consequence for serving. With a capacity limit, whether a token is dropped depends on the other tokens in its batch, so the same prompt can produce different outputs depending on who else is being served. Even without capacity, batch-dependent kernel choices can change floating-point results (§6.4 discusses batch invariance). Dropless routing, as in DeepSeek-V3, removes the first source of that variation.

What experts actually specialize in

The name invites a story. Expert 3 knows chemistry, expert 5 speaks French, expert 7 does arithmetic. The evidence says something narrower and more interesting.

Experts specialize by token and syntax more than by topic. In ST-MoE's encoder, experts specialized in punctuation, verbs, proper names, numbers, and the sentinel tokens of the pretraining objective. In Mixtral, expert assignments for ArXiv papers, PubMed abstracts, and PhilPapers were very similar at every layer, with only DeepMind Mathematics, a synthetic dataset, standing out a little. What Mixtral's router did follow was syntax, for example the Python keyword self and indentation tokens going consistently to the same experts.
Not by language. ST-MoE's multilingual model routed English, French, Japanese, and Chinese tokens without separating them by language.
Less in decoders. In ST-MoE the decoder's routing was close to uniform for sentinel tokens (entropy about 3.4 against a uniform 3.5), where the encoder's was far more concentrated (about 1.2 to 2.2).
More with more, smaller experts and gentler balancing. OLMoE, with 64 small experts, found an expert in its first layer that took almost only arXiv text, and experts tied to non-Latin scripts, brackets and quotes, weekdays and holidays, and country names. DeepSeek-V3's ablation found that bias balancing produced more domain specialization than an auxiliary loss, which forces each batch to spread evenly.
And it may not be necessary. Hash layers (Roller et al. 2021) route each token by a fixed hash of its id, with no learning at all, and still matched or beat Switch and BASE routing on language modeling. Much of MoE's benefit comes from having more parameters, not from clever assignment.

So the honest summary is that specialization is real, mostly lexical and syntactic, varies by layer, grows with finer experts, and is not organized by human topics. Measuring it is a small computation. Count how often tokens from each domain go to each expert, and compute the mutual information between domain and expert (§0.5).

Python · cell 3

Core

How specialized are the experts?

Implement expert_specialization(counts) for a count matrix of shape (n_domains, E), where counts[j, i] is how many tokens of domain $j$ were routed to expert $i$ (top-1, for simplicity). Return the mutual information $I(\text{domain};\text{expert})$ in nats divided by $H(\text{domain})$ , both computed from the joint distribution the counts define, so 0 means routing ignores the domain and 1 means the expert identifies the domain exactly. Treat $0\log 0$ as 0.

import numpy as np

def expert_specialization(counts):
    p = counts / counts.sum()
    return float(p.max())                 # placeholder

Hidden tests, hints, and AI help, from $1.99 a month.

In PyTorch

The arithmetic of the memory and batch trade-off is worth having as a function you can run on any configuration. Here it compares Mixtral with the two dense models that bracket it, a dense model of its active size and one of its total size, at several decoding batch sizes, counting bytes of weights read per generated token in bfloat16.

🔥 In PyTorchfor a machine with PyTorch

At $B = 1$ the MoE reads as little as a dense model of its active size. By $B = 16$ it reads nearly as much as a dense model of its total size, while still doing the arithmetic of the active size. Only at large batches does the comparison shift back to compute, where the MoE wins by its full sparsity factor.

When is MoE worth it?

situation	MoE?	why
pretraining with a fixed compute budget, at scale	yes	more quality per FLOP, the gap grows with budget (§13.5)
serving many users on a large cluster	yes	big batches amortize expert reads, cost per token follows active parameters
one user, model fits in fast memory	often yes	batch 1 reads only $k$ experts per layer
small batches on memory-limited hardware	often no	reads nearly all weights at moderate batch, all must fit somewhere
fine-tuning on a small single-task dataset	with care	overfits faster, needs different hyperparameters, freeze experts
a small team that needs simple, robust training	maybe not	routing, balancing, all-to-all, and stability add real complexity
fixed total memory, maximum quality	usually dense	a dense model uses every stored parameter on every token

The last row deserves a sentence of its own. At equal active parameters, an MoE beats a dense model (every comparison in this chapter, from Switch to DeepSeekMoE). At equal total parameters, a dense model generally beats the MoE, because it uses all its parameters on every token at a much higher compute cost. An MoE's quality falls between the two, and Clark et al.'s effective parameter count (§13.5) is a way to say where.

🎯 Interview questions on this section

Why are MoE models harder to train stably, and what fixes it?

Router logits grow and bfloat16 rounding erases their differences, exponentials amplify roundoff, and routing is discontinuous, so small input changes flip expert assignments and the output jumps. Fixes are the router z-loss (in ST-MoE it made all runs stable and slightly improved quality), computing the router in float32 (Switch, which turned a diverging bfloat16 run into one matching float32), smaller initialization (a tenth of the usual scale), and balancing. Many other stabilizers, such as update clipping or removing multiplicative gates, also stabilize but cost quality.

What is different about fine-tuning an MoE model?

It overfits small datasets faster, so use more dropout inside experts (Switch used 0.4). It prefers smaller batches and higher learning rates than a dense model (ST-MoE). Freezing the experts and updating only the shared parameters works about as well as full fine-tuning, while updating only experts does worse. MoE models benefit more from broad instruction tuning than dense ones (Shen et al. 2023). And routing can drift under new data, so watch expert load during fine-tuning.

How many distinct experts does a decoding step touch, and why does it matter?

With $B$ tokens choosing $k$ of $E$ experts uniformly, $\mathbb{E}[U] = E(1 - (1-k/E)^B)$ . Decoding is memory-bound, so bytes read set the speed. At $B = 1$ an MoE reads only $k$ experts, like a dense model of its active size. At moderate $B$ , $U$ approaches $E$ while each expert serves few tokens, so the MoE reads almost all its weights for the arithmetic of its active size. Mixtral touches 7.9 of 8 experts at $B = 16$ with about 4 tokens each. Large batches and wide expert parallelism are needed before compute dominates.

Do experts specialize in topics?

Mostly not. Mixtral's routing barely differs between ArXiv, PubMed, and PhilPapers, and follows syntax instead (Python's self, indentation). ST-MoE's encoder experts specialize in token types (punctuation, verbs, names, numbers), not languages, and its decoder shows little specialization. Finer experts and bias balancing increase domain specialization (OLMoE, DeepSeek-V3), and hash routing with no learning is competitive, so much of the gain comes from parameter count rather than semantic assignment.

When would you choose a dense model over an MoE?

When memory, not compute, is the constraint, as in single-GPU or edge deployment at moderate batch sizes, where an MoE must store every expert and reads most of them each step. When the team needs simple, robust training without routing, balancing, and all-to-all. When fine-tuning on small single-task data. And when the goal is maximum quality at a fixed parameter count, since a dense model uses all its parameters on every token. MoE wins when compute per token is the constraint, in pretraining and in high-throughput serving.

How does batch size affect MoE training and serving?

Each expert sees about $kB/E$ of a batch (Shazeer's shrinking batch problem), so MoE needs large global batches and expert parallelism to keep expert matmuls efficient. Larger routing groups also reduce capacity overflow, since load fluctuations shrink relative to the mean. In serving, small batches are memory-bound on many experts, and with capacity limits a request's output depends on what else is in its batch, which dropless routing avoids.

🧭 What to remember

MoE training is less stable because routing is discontinuous and router logits suffer in low precision. The z-loss, float32 routers, and small initialization are the fixes that do not cost quality.
Fine-tuning differs. Sparse models overfit small tasks, prefer smaller batches and higher learning rates, can freeze their experts, and gain more from instruction tuning than dense models.
Memory follows total parameters, compute follows active ones. A decoding step touches $E(1-(1-k/E)^B)$ experts, so batch 1 is cheap, moderate batches read nearly everything, and only large batches make the arithmetic pay for the bytes.
Experts specialize mainly by token type and syntax, not by topic or language, and fixed hash routing is competitive, so do not anthropomorphize them.
Choose MoE when compute per token is the constraint (pretraining, high-throughput serving), and dense when memory, simplicity, or small-data fine-tuning dominate.
Next, Part IV turns from building base models to shaping them, starting with what post-training does.

Zoph et al. (2022). ST-MoE: Designing Stable and Transferable Sparse Expert Models. Stability, fine-tuning, and specialization, in one careful study.
Shen et al. (2023). Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models. Why MoE models need instruction tuning more than dense ones.
Jiang et al. (2024). Mixtral of Experts. Section 5 analyzes routing by domain, syntax, and position.
Muennighoff et al. (2024). OLMoE: Open Mixture-of-Experts Language Models. Router saturation, co-activation, and domain and vocabulary specialization.
Roller et al. (2021). Hash Layers For Large Sparse Models. Routing with no learning, and what it says about specialization.
Fedus, Zoph & Shazeer (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Selective precision, initialization, expert dropout, and distillation.
Zheng et al. (2025). Group Sequence Policy Optimization. Why MoE routing makes token-level RL objectives unstable, and a sequence-level fix.

Zero to Frontier is made for larger screens

When MoE is worth it

Training stability

Router saturation between checkpoints

Fine-tuning

Memory versus compute at inference

Experts touched by a decoding step

Batch size, in training and serving

What experts actually specialize in

How specialized are the experts?

In PyTorch

When is MoE worth it?

Read more

Watch next