In 2021 the largest MoE language model had 2,048 experts per layer and sent each token to one of them. In late 2024 DeepSeek-V3 had 256 experts per layer, sent each token to eight of them plus one more that every token shares, and made each expert one seventh the width of Mixtral's. Why did the designs move toward many small experts used several at a time, and why did a shared expert appear? Each change answered a problem that the previous generation ran into, and following them in order is the fastest way to understand what a modern MoE is and why.

This section walks that path. GShard and Switch made MoE work in Transformers. GLaM and ST-MoE showed it paid off and made it stable. Mixtral made it the default for open models. DeepSeekMoE split experts finely and added shared ones, and DeepSeek-V3 combined everything with the bias balancing of §13.3. Then we look at two ways to get an MoE without starting from scratch or guessing, upcycling a dense checkpoint and MoE scaling laws.

GShard and Switch, the first Transformer MoEs

GShard (Lepikhin et al. 2020) replaced every other FFN of a Transformer translation model with an MoE layer and scaled it to 600 billion parameters, with 2,048 experts per MoE layer and 36 layers, trained on 2,048 TPU v3 cores for four days to translate 100 languages into English. Its router was top-2 with normalized gates, its capacity was about $2N/(GE)$ per expert for $N$ tokens split into $G$ groups, and it had a cheap exploration trick. The second expert is used only with probability proportional to its gate, dispatch happens if $2g_2 > u$ for $u$ uniform on $[0, 1]$, so a token with a weak second choice usually skips it and saves capacity. GShard's other contribution was the compiler support for sharding, which made expert parallelism (§13.4) a few annotations instead of a rewrite.

The Switch Transformer (Fedus, Zoph & Shazeer 2022) asked how simple an MoE could be. It used top-1 routing with the unnormalized softmax probability as the gate (§13.2), a capacity factor of 1.0 to 1.25, and the auxiliary loss of §13.3 with $\alpha = 10^{-2}$. Three training details made it stable and are still standard.

• Selective precision. The router runs in float32 while the rest of the model uses bfloat16. A Switch-Base model diverged in pure bfloat16 and matched float32 quality with the router alone in float32, at bfloat16 speed.
• Smaller initialization. Weights were initialized with one tenth of the usual scale, which cut the variance of results across seeds sharply.
• Expert dropout. For fine-tuning, dropout of 0.4 inside experts and 0.1 elsewhere worked better than uniform dropout, because the experts overfit first.

The headline results were a pretraining speedup of about 7x over T5-Base at the same FLOPs per token (Switch-Base with 64 experts reached T5-Base's quality in about one seventh of the time), and a 1.6-trillion-parameter Switch-C with 2,048 experts. Switch also showed that distilling a sparse model into a dense one kept about 30% of the sparse model's quality gain while cutting parameters by up to 99%.

GLaM and ST-MoE, MoE at GPT-3 scale

GLaM (Du et al. 2022) was the first decoder-only MoE compared head to head with GPT-3. It had 1.2 trillion parameters in 64 layers, with 64 experts and top-2 routing in every other layer, and activated 96.6 billion parameters per token, 8% of the total. Averaged over 29 benchmarks it beat GPT-3 in zero-, one-, and few-shot settings, while its training used about a third of GPT-3's energy (456 MWh against 1,287 MWh) and its inference about half the FLOPs per token (180 billion against 350 billion). That was the first strong evidence that sparse models were not just a translation curiosity.

ST-MoE (Zoph et al. 2022) set out to make sparse models reliable. Its 269-billion-parameter model, with 64 experts in every fourth layer, cost about as much per token as a 32B dense model. It introduced the router z-loss, compared many stabilization methods (§13.3 and §13.6), and studied fine-tuning in detail. Its practical recipe of top-2 routing, a capacity factor of 1.25 in training, and at most one expert per accelerator core became the default for a while.

Mixtral, the open default

Mixtral 8x7B (Jiang et al. 2024) took Mistral 7B and replaced each FFN with eight SwiGLU experts of the same width, $d_e = 14{,}336$, routed top-2 in every layer with a softmax over the two chosen logits. Its 46.7B total and 12.9B active parameters are counted in §13.1. It matched or beat Llama 2 70B and GPT-3.5 on most of the benchmarks reported, with particularly large margins in mathematics, code, and multilingual tasks, at about a fifth of Llama 2 70B's compute per token. Mixtral's simplicity, eight big experts and top-2, is the "classic" MoE that most people picture.

DeepSeekMoE, finer experts and shared ones

Dai et al. (2024) asked what an expert should be. With 8 or 16 large experts and top-2 routing, they argued, each expert has to cover many unrelated kinds of tokens, so experts end up knowing overlapping things. They made two changes, each keeping the parameters and FLOPs fixed.

Fine-grained expert segmentation. Split each of the $E$ experts into $m$ experts of width $d_e/m$, giving $mE$ experts, and route each token to $mk$ of them. The parameter count is unchanged, $mE\cdot 3d(d_e/m) = 3dd_eE$, and so is the compute per token, $mk\cdot 6d(d_e/m) = 6dd_ek$. What changes is the number of ways a token can combine experts,

In plain words, cut every expert into smaller pieces and let each token pick more of them. The same budget buys far more flexible mixtures, so each piece can specialize more narrowly.

Shared expert isolation. Make $E_s$ of the experts shared, always active for every token, outside the router. With $mE$ fine-grained experts in total, $mE - E_s$ remain routed, and each token picks $mk - E_s$ of them, so the compute stays the same. The shared experts learn what every token needs (common syntax, frequent words), so the routed experts do not each have to store a copy.

The evidence came from controlled comparisons. DeepSeekMoE 2B matched GShard 2.9B, a model with 1.5 times its expert parameters and 1.5 times its expert computation. DeepSeekMoE 16B (2 shared and 64 routed experts, each a quarter of the standard width, 2 shared plus 6 routed active, 16.4B total, 2.8B active) matched LLaMA2 7B with about 40% of its compute. And in the 2B model, disabling the shared expert and activating one more routed expert in its place raised the Pile loss from 1.808 to 2.414, so the shared expert holds something the routed ones do not.

The evidence is not unanimous. OLMoE (Muennighoff et al. 2024) also found large gains from finer experts (at fixed active and total parameters, going from 8 experts with 1 active to 32 with 4 active improved HellaSwag and MMLU by about 10%, and 64 with 8 active added 1 to 2% more) but found a shared expert slightly worse than one more routed expert, and left it out.

from math import comb

def fine_grain(E, k, d_e, d, m, n_shared):
    return {"routed": E * m, "active_routed": k * m, "d_e": d_e // m,
            "params": 3 * d * d_e * E, "active_params": 3 * d * d_e * k, "combos": comb(E, k)}

DeepSeek-V2 and V3

DeepSeek-V2 (DeepSeek-AI 2024a) scaled the design to 236B total and 21B active parameters, with 2 shared and 160 routed experts and 6 routed experts per token, multi-head latent attention (§12.5), a limit on how many devices a token's experts could span, and three auxiliary balancing losses (per expert, per device, and for communication). It reported 42.5% lower training cost than the dense DeepSeek 67B.

DeepSeek-V3 (DeepSeek-AI 2024b) is the recipe most new MoE models now start from. Here it is in one place, with the section of this chapter that explains each part.

Why a sigmoid instead of a softmax? The report does not say, but the reasoning is easy to reconstruct. With a softmax over 256 experts, every expert's probability depends on every other expert's logit, so pushing one expert's probability up pushes all others down. A sigmoid gives each expert an independent affinity in $(0, 1)$, which suits the bias method (each bias shifts one expert's score without disturbing the rest) and fine-grained experts. Wang et al. (2024) found sigmoid gating slightly better than softmax in their experiments, and bias balancing works with either.

import numpy as np

def deepseek_moe(X, W_r, b, routed, shared, k, scale):
    raise NotImplementedError

The current generation

The table collects the configurations of well-documented MoE models. Counts for the last rows are computed from the released configurations, the way §13.1 counts Mixtral.

Read down the table and the trends are plain. Experts multiplied and shrank, from 8 wide ones to hundreds of narrow ones. The number per token rose from 1 or 2 to 8. And the ratio of total to active parameters grew, from about 4 for Mixtral to 18 for DeepSeek-V3, 31 for Kimi K2, and 23 for gpt-oss-120b. Sparsity itself became a scaling axis.

Sparse upcycling

Training an MoE from scratch is expensive, and many teams already own a good dense checkpoint. Komatsuzaki et al. (2023) proposed sparse upcycling. Copy every parameter of the dense model, replace some FFNs with MoE layers whose experts are all initialized as copies of the original FFN, initialize the new routers randomly (they used a normal distribution with standard deviation 0.02), and continue training.

There is a neat reason this starts well. If all experts are identical copies of the dense FFN and the gates of each token sum to 1, then

whatever the router does. In plain words, at the moment of upcycling the MoE computes exactly the dense function, as long as the gates are normalized and no token is dropped. Training then breaks the symmetry, because different experts see different tokens and receive different gradients. With unnormalized gates (Switch's $p_{t,i}$) the output is scaled by $p_{t,i} < 1$ and the function changes immediately, one source of the initial quality drop the paper reports.

Their setup upcycled half the MLP layers (every other one) into 32 experts, used expert-choice routing in vision models and the T5 encoder and top-2 routing in the decoder, with a capacity factor of 2. Upcycled T5-Base and T5-Large beat their dense counterparts by 1.5 to 2 SuperGLUE points using 55% and 46% extra training. For ViT-B/16, gaining 1% on 10-shot ImageNet took 13% extra training time with upcycling against 58% for continued dense training. And an MoE trained from scratch needed about 120% of the original dense model's compute to catch up with the upcycled one. OLMoE later found a from-scratch MoE caught up with an upcycled OLMo-1B after about 25% of the dense model's budget (500B of its 2T tokens), so the advantage depends on the setting, and is largest when the extra budget is small.

import numpy as np

def upcycle(dense, E, d, rng, std=0.02):
    return [dense] * E, np.zeros((d, E))

Scaling laws for MoE

Dense scaling laws (§9.3) relate loss to parameters and tokens. For MoE there is a third knob, the number of experts, and a fourth, their size. Two papers frame most of what is known.

Clark et al. (2022) trained routed models across sizes and expert counts on a fixed 130B tokens and fit, for a dense-equivalent size $N$ (the parameters a token uses) and $E$ experts,

with logarithms to base 10 (an exception to the book's natural-log convention, kept so the fitted constants match the paper), where $\hat E$ is $E$ passed through a saturating transform so that very large expert counts stop helping. For their best router (a Sinkhorn-balanced method, S-BASE) they fit $a = -0.082$, $b = -0.108$, $c = 0.009$, $d = 1.104$. In plain words, more experts lower the loss, but the positive interaction term $c$ means the benefit of experts shrinks as the base model grows. The slope with respect to $\log\hat E$ is $b + c\log N$, which reaches zero at $\log N = -b/c = 12$, so $N \approx 10^{12}$ with these rounded constants. The paper's cutoff for S-BASE is 937 billion parameters. They recommended 64 to 128 experts, and defined an effective parameter count, the size of the dense model with the same predicted loss.

Krajewski et al. (2024) argued that this cutoff was an artifact of holding the token count fixed. They added granularity $G$, the ratio of the dense FFN width to an expert's width (DeepSeekMoE's $m$ in this section), varied tokens and granularity together, and fit

with $N$ the MoE's non-embedding parameters, $D$ the training tokens, an expansion rate of 64 (each MoE layer holding 64 times a dense FFN's parameters), and fitted values $a = 18.1$, $\alpha = 0.115$, $b = 30.8$, $\beta = 0.147$, $g = 2.1$, $\gamma = 0.58$, $c = 0.47$. In plain words, finer experts shrink the parameter term, with diminishing returns as $G^{-0.58}$. Their conclusions were that $G = 1$ (experts as wide as the dense FFN, the Mixtral design) is suboptimal at almost any budget, that the compute-optimal granularity grows with the budget, and that with tokens and granularity tuned, MoE beats dense at every scale they studied, with a gap that widens as compute grows.

Granularity is not free, and the paper accounts for it. The router costs FLOPs proportional to the number of experts, which for a fixed expansion rate grows with $G$. And §13.4 showed that narrower experts mean fewer FLOPs per byte of all-to-all traffic. The cell puts numbers on both sides for a model with $d = 4096$, using the paper's routing cost model ($d\cdot E\cdot G\cdot c_r$ router FLOPs per token per layer with $c_r = 14$, against $12d^2c_f$ with $c_f = 6$ for the rest of a block).

Going from $G = 1$ to $G = 64$ shrinks the parameter term by about 10%, a large gain in a scaling law, while the router's share of compute climbs past 19% and the communication intensity falls by 64x. The best granularity in practice balances all three, which is roughly where DeepSeek-V3 ($d_e = 2048$ against a dense width of 18,432, so $G = 9$) and Qwen3 (1,536 against 12,288, $G = 8$) landed.

In PyTorch

Here is the DeepSeek-V3 layer as a PyTorch module, with sigmoid affinities, the selection bias stored as a buffer (so it never receives gradients), normalized and scaled gates, a shared expert, and the bias update as a separate no-gradient step called after each optimizer step.

Read more

• Fedus, Zoph & Shazeer (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. The simplest strong MoE, with all its training details.
• Du et al. (2022). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. The decoder-only MoE that matched GPT-3 at a third of the energy.
• Dai et al. (2024). DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models. Fine-grained and shared experts, with careful ablations.
• DeepSeek-AI (2024). DeepSeek-V3 Technical Report. The full modern recipe.
• Komatsuzaki et al. (2023). Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints. How to turn a dense model into an MoE, and when it pays.
• Krajewski et al. (2024). Scaling Laws for Fine-Grained Mixture of Experts. Granularity as a scaling variable, and the case against wide experts.
• Muennighoff et al. (2024). OLMoE: Open Mixture-of-Experts Language Models. A fully open MoE with ablations of granularity, shared experts, upcycling, and routing.

Watch next

▶

▶

▶