How frontier models store hundreds of billions of parameters and use a few billion per token, from the router's gradient to the all-to-all traffic, the balancing tricks, and the designs behind Mixtral and DeepSeek-V3.

Chapter 13 · Section 13.1 · 50 min

Conditional computation

Here is a puzzle about the model you built in Chapter 6. When it reads the token "the", every one of its parameters takes part. The weights that, in a trained model, store facts about protein folding, Python syntax, and the kings of France are all multiplied through, and almost all of that work is wasted on a token that needs none of it. Now suppose you want a model that knows eight times as much. A dense Transformer gives you exactly one way to get it, eight times the parameters, and with them eight times the arithmetic for every token, forever, in training and in serving.

Is there a way to have the parameters without paying for them on every token? That question is conditional computation, and the answer that took over frontier language models is the mixture of experts (MoE). Mixtral 8x7B stores 46.7 billion parameters and uses 12.9 billion of them per token. DeepSeek-V3 stores 671 billion and uses about 37 billion. By the end of this section you will be able to derive both pairs of numbers on a whiteboard, say exactly what an MoE layer computes, explain why it is not an ensemble, and trace the idea from a 1991 paper on vowel recognition to today's largest open models.

Parameters and FLOPs are welded together in a dense model

Recall the 6N rule from §5.6. Every weight in a matrix multiplication does one multiply and one add per token in the forward pass, so a model with $N$ such weights costs about $2N$ FLOPs per token forward and $6N$ to train, and $6ND$ for $D$ training tokens. The rule has no slack in it. Each parameter is touched by each token, so parameters and compute are welded together.

That would not matter if parameters were cheap in some other way, but they are what a model knows. In §5.3 we read the feed-forward network as a key-value memory, with each hidden unit a key that fires on some pattern and writes a value back into the stream, and the FFNs hold about two thirds of each block's weights. Scaling laws (§9.2) say loss keeps falling as parameters grow. So we want more of them, and the dense recipe charges us for every one of them on every token.

Conditional computation breaks the weld. Keep many FFNs, and let each token use only a few of them, chosen by a small learned function of the token itself. The parameter count, and with it the capacity to store knowledge, grows with the number of FFNs. The compute per token grows only with the number used.

The MoE layer

An MoE layer replaces the FFN sublayer of a Transformer block. Attention, the norms, and the residual stream stay exactly as they were. For one token's input $\mathbf{x} \in \R^{d}$ (a row, the output of the pre-FFN norm), the layer has

$E$ experts, each an ordinary FFN of the kind in §5.3. In modern models each is a SwiGLU network $\operatorname{FFN}_i(\mathbf{x}) = \big(\operatorname{SiLU}(\mathbf{x}\mathbf{W}_g^{(i)}) \odot \mathbf{x}\mathbf{W}_u^{(i)}\big)\mathbf{W}_d^{(i)}$ with $\mathbf{W}_g^{(i)}, \mathbf{W}_u^{(i)} \in \R^{d\times d_e}$ and $\mathbf{W}_d^{(i)} \in \R^{d_e\times d}$ , where $d_e$ is the expert's hidden width.
a router (also called the gate), a linear map $\mathbf{W}_r \in \R^{d\times E}$ that gives one logit per expert, $\mathbf{r} = \mathbf{x}\mathbf{W}_r \in \R^{E}$ .
a number $k$ of experts to use per token, usually 1, 2, or 8.

The router picks the $k$ experts with the largest logits, the set $\mathcal{K} = \operatorname{TopK}(\mathbf{r}, k) \subset [E]$ , turns their logits into weights $g_i$ that sum to one, and the layer returns the weighted sum of just those experts' outputs,

g_i = \frac{e^{r_i}}{\sum_{j \in \mathcal{K}} e^{r_j}} \ \text{ for } i \in \mathcal{K}, \qquad g_i = 0 \ \text{ otherwise}, \qquad \operatorname{MoE}(\mathbf{x}) = \sum_{i \in \mathcal{K}} g_i\, \operatorname{FFN}_i(\mathbf{x}).

In plain words, score every expert with one cheap matrix product, keep the best $k$ , run only those, and blend their answers in proportion to how strongly the router preferred them.

The block is otherwise unchanged, $\mathbf{h} = \mathbf{x}_{\text{in}} + \operatorname{MoE}(\operatorname{RMSNorm}(\mathbf{x}_{\text{in}}))$ . Two details matter from the start. The experts that are not chosen are never evaluated, which is the whole point. And the router itself is evaluated for all $E$ experts, which costs $2dE$ FLOPs per token, tiny next to an expert (about $6d\,d_e$ ) but not zero. There are several ways to turn router logits into weights (softmax over all experts, softmax over the chosen ones, sigmoids), and they behave differently in training. §13.2 takes them apart. Here we use the version above, which is Mixtral's.

A worked example small enough to check by hand

Take $d = 4$ , $E = 4$ , $k = 2$ , and a router whose logits for one token come out as $\mathbf{r} = (2.0, 0.5, 1.0, -1.0)$ . The top two are experts 0 and 2. Their weights are a two-way softmax,

g_0 = \frac{e^{2}}{e^{2} + e^{1}} = \frac{1}{1 + e^{-1}} = 0.7311, \qquad g_2 = 0.2689,

and $g_1 = g_3 = 0$ . The layer runs experts 0 and 2 and returns $0.7311\,\operatorname{FFN}_0(\mathbf{x}) + 0.2689\,\operatorname{FFN}_2(\mathbf{x})$ . The cell below does exactly this, then checks it against the wasteful version that runs all four experts and multiplies the unused ones by zero.

Python · cell 1

Total and active parameters

Two counts describe an MoE model, and confusing them is the most common mistake people make about these models.

The total parameters $N_{\text{total}}$ are everything stored. They set the memory needed to hold the model.
The active parameters $N_{\text{active}}$ are the ones a single token actually multiplies through. They set the FLOPs per token, so they play the role of $N$ in the 6N rule.

For one MoE layer with $E$ SwiGLU experts of width $d_e$ , a bias-free router, and $k$ experts per token,

N^{\text{layer}}_{\text{total}} = 3\,d\,d_e\,E + d\,E, \qquad N^{\text{layer}}_{\text{active}} = 3\,d\,d_e\,k + d\,E .

In plain words, storage grows with the number of experts, compute grows with the number used, and the router is the only part every token pays for in full.

Mixtral 8x7B, counted exactly

Mixtral (Jiang et al. 2024) is Mistral 7B with each FFN replaced by eight copies of the same shape and top-2 routing. Its published configuration is $d = 4096$ , $L = 32$ layers, $h = 32$ query heads and $n_{kv} = 8$ key-value heads of width $d_k = 128$ , $d_e = 14{,}336$ , $E = 8$ , $k = 2$ , a vocabulary of $V = 32{,}000$ , and separate (untied) input and output embeddings. Count one layer.

Attention with grouped-query attention (§4.5). $\mathbf{W}_Q$ and $\mathbf{W}_O$ are $4096\times4096$ (since $h d_k = 4096$ ), and $\mathbf{W}_K$ , $\mathbf{W}_V$ are $4096 \times 1024$ (since $n_{kv} d_k = 1024$ ), so $2\cdot 4096^2 + 2\cdot 4096\cdot 1024 = 41{,}943{,}040$ .
One expert. Three matrices of $4096 \times 14{,}336$ , so $176{,}160{,}768$ . Eight of them are $1{,}409{,}286{,}144$ .
Router. $4096 \times 8 = 32{,}768$ .
Two RMSNorm gains. $8{,}192$ .

That is $1{,}451{,}270{,}144$ per layer and $46{,}440{,}644{,}608$ for 32 layers. Add the input and output embeddings, $2 \times 32{,}000 \times 4096 = 262{,}144{,}000$ , and the final norm's $4{,}096$ , and the total is

N_{\text{total}} = 46{,}702{,}792{,}704 \approx 46.7\text{B}.

For the active count, replace eight experts by two in every layer, $2 \times 176{,}160{,}768 = 352{,}321{,}536$ , which gives $394{,}305{,}536$ per layer and

N_{\text{active}} = 32 \times 394{,}305{,}536 + 262{,}144{,}000 + 4{,}096 = 12{,}879{,}925{,}248 \approx 12.9\text{B},

counting both embedding matrices in full. (Leave out the input embedding, which is a lookup and costs no FLOPs, and it is 12.7B. Both round to the paper's "13B active".)

Why is "8x7B" not 56 billion? Because only the FFNs were multiplied by eight. Mistral 7B has 7.24B parameters, of which 1.60B are attention, norms, and embeddings, and those are shared by all experts. Eight copies of the 5.64B of FFN weights plus one copy of the shared 1.60B is 46.7B.

DeepSeek-V3, counted exactly

DeepSeek-V3 (DeepSeek-AI 2024) has $d = 7168$ and $L = 61$ layers. The first three layers keep a dense SwiGLU FFN of width 18,432. The other 58 are MoE layers with $E = 256$ routed experts of width $d_e = 2048$ , $k = 8$ of them per token, plus one shared expert of the same width that every token always uses (§13.5 explains why). Attention is multi-head latent attention (§12.5), whose projections, low-rank factors, and two small norms add up to $187{,}107{,}328$ per layer with $h = 128$ heads. (That is a $7168\times1536$ query down-projection, a $1536\times(128\cdot192)$ query up-projection, since each head's query has 128 content and 64 rotary dimensions, a $7168\times576$ key-value down-projection that includes the shared rotary key, a $512\times(128\cdot256)$ up-projection to the keys and values, a $(128\cdot128)\times7168$ output projection $\mathbf{W}_O$ , and norms of sizes 1,536 and 512.) The vocabulary is $V = 129{,}280$ , untied.

One expert. $3 \times 7168 \times 2048 = 44{,}040{,}192$ .
An MoE layer. Attention $187{,}107{,}328$ , plus $257$ experts $11{,}318{,}329{,}344$ , plus the router $256\times7168 = 1{,}835{,}008$ , plus two norms $14{,}336$ , is $11{,}507{,}286{,}016$ .
A dense layer. Attention plus $3\times7168\times18{,}432 = 396{,}361{,}728$ plus norms, $583{,}483{,}392$ .

So $N_{\text{total}} = 3 \times 583{,}483{,}392 + 58 \times 11{,}507{,}286{,}016 + 2\times129{,}280\times7168 + 7168 = 671{,}026{,}404{,}352$ , the paper's 671B. Over 97% of it is routed experts. Per token, an MoE layer uses attention, the router, the norms, and nine experts (eight routed and the shared one), $585{,}318{,}400$ parameters, so

N_{\text{active}} = 3\times583{,}483{,}392 + 58\times585{,}318{,}400 + 1{,}853{,}358{,}080 + 7{,}168 = 37{,}552{,}282{,}624,

or 36.6B without the input embedding, the paper's "37B activated". (The released checkpoint also contains a multi-token prediction module used in training, which the 671B leaves out.)

Python · cell 2

Interview

Total and active parameters of an MoE model

Implement moe_param_count(cfg) returning the pair (total, active) for a decoder whose config is a dict with keys V (vocabulary), d (width), L (layers), attn (attention parameters per layer, given), n_dense (how many of the first layers keep a dense FFN), d_ff (that dense FFN's hidden width), E (routed experts per MoE layer), E_shared (shared experts, always active), k (routed experts per token), d_e (expert hidden width), and tied (whether input and output embeddings are one matrix). Every FFN and expert is a bias-free SwiGLU, every layer has two RMSNorm gains of size $d$ , there is one final norm, and the router is a bias-free $d\times E$ matrix. Count the embedding matrices in full in both numbers.

Interviewers ask this as "how many parameters does Mixtral have, and how many does a token use?", and the tests check exactly that, plus DeepSeek-V3.

def moe_param_count(cfg):
    # placeholder: counts every expert as active
    d, L = cfg["d"], cfg["L"]
    per_layer = cfg["attn"] + 3 * d * cfg["d_e"] * cfg["E"]
    total = L * per_layer + 2 * cfg["V"] * d
    return total, total

Hidden tests, hints, and AI help, from $1.99 a month.

FLOPs per token, and the 6N rule for MoE

Because the unchosen experts never run, the 6N rule carries over with $N$ replaced by the active matrix parameters. Training on $D$ tokens costs

C_{\text{train}} \approx 6\,N_{\text{active}}\,D \quad (\text{plus attention's } 12LTd \text{ per token, §5.6}).

In plain words, an MoE model trains at the price of a dense model the size of its active parameters, while storing many times more.

DeepSeek-V3 was pretrained on 14.8 trillion tokens. With $N_{\text{active}} \approx 37\times10^{9}$ that is $6 \times 37\times10^{9}\times14.8\times10^{12} \approx 3.3\times10^{24}$ FLOPs. A dense model with all 671B parameters would have needed $6.0\times10^{25}$ , eighteen times more. The paper reports 2.788 million H800 GPU hours for the whole run, 2.664 million of them for pretraining, which §8.9 turns into a model FLOPs utilization (the measure of §6.7) of about 35 percent of the BF16 peak.

Two caveats keep this honest. First, the router's $2dE$ FLOPs per token are real but small. For DeepSeek-V3 they are $2\times 7168\times 256 \approx 3.7$ million per layer against about 793 million for the nine experts, half a percent. Second, FLOPs are not time. An MoE layer has to move tokens to their experts, often across GPUs, and run many small matrix products instead of one big one. Those costs are the subject of §13.4, and they are why an MoE model with the FLOPs of a 37B dense model does not train quite as fast as one.

Warm-up

Training compute of a dense model and its MoE counterpart

Implement train_flops(n_active, tokens) returning $6\,N_{\text{active}}\,D$ as a float, and flops_ratio(n_total, n_active) returning how many times more compute a dense model with $N_{\text{total}}$ parameters would need for the same tokens.

def train_flops(n_active, tokens):
    return 2.0 * n_active * tokens        # placeholder: forward pass only

def flops_ratio(n_total, n_active):
    return 1.0

Hidden tests, hints, and AI help, from $1.99 a month.

An MoE layer is a wide FFN with blocks switched off

It is tempting to picture an MoE model as eight small models voting. That picture is wrong in a way that matters, and the right picture comes straight from §5.3.

An FFN is a sum over its hidden units. For a ReLU network, $\operatorname{FFN}(\mathbf{x}) = \sum_{j=1}^{d_{\text{ff}}} \phi(\mathbf{x}\,\mathbf{W}_{1,:,j})\,\mathbf{W}_{2,j,:}$ , one key-value memory per unit. Cut the $d_{\text{ff}}$ hidden units into $E$ consecutive blocks of $d_e = d_{\text{ff}}/E$ , and the sum splits into $E$ smaller FFNs,

\operatorname{FFN}_{\text{wide}}(\mathbf{x}) = \sum_{i=1}^{E} \operatorname{FFN}_i(\mathbf{x}),

where $\operatorname{FFN}_i$ uses only block $i$ 's columns of $\mathbf{W}_1$ and rows of $\mathbf{W}_2$ . The same holds for SwiGLU, block by block. So an MoE layer is a single FFN of width $E\,d_e$ in which, for each token, all but $k$ blocks of hidden units are switched off and the surviving blocks are rescaled by their gates. In plain words, the experts are slices of one very wide memory, and the router decides which slices each token may read.

This explains three things. It is why the experts share everything else (attention, embeddings, the residual stream). It is why routing happens independently in every layer, with no notion of "the model's expert 3". A token in Mixtral picks one of $\binom{8}{2} = 28$ expert pairs in each of 32 layers, so there are $28^{32} \approx 2\times10^{46}$ different paths through the network, which is nothing like an ensemble of eight models. And it is why an MoE model can be built from a dense one by copying its FFN into every expert (upcycling, §13.5), since that is a wide FFN with redundant blocks.

Python · cell 3

Core

Cut a wide FFN into experts

Implement split_into_experts(W1, W2, E) that takes a ReLU FFN with $\mathbf{W}_1 \in \R^{d\times d_{\text{ff}}}$ and $\mathbf{W}_2 \in \R^{d_{\text{ff}}\times d}$ and returns a list of $E$ pairs (A_i, B_i) with A_i of shape (d, d_ff // E) and B_i of shape (d_ff // E, d), so that summing relu(X @ A_i) @ B_i over all experts reproduces the wide FFN. Expert $i$ owns hidden units $i\,d_e$ to $(i+1)\,d_e - 1$ . Raise ValueError if $E$ does not divide $d_{\text{ff}}$ .

import numpy as np

def split_into_experts(W1, W2, E):
    raise NotImplementedError

Hidden tests, hints, and AI help, from $1.99 a month.

Where the idea came from

📜 From vowels to trillion-parameter models

Jacobs, Jordan, Nowlan, and Hinton (1991) introduced adaptive mixtures of local experts, a set of small networks plus a gating network that decides which expert handles each input, and showed that it split a speaker-independent vowel recognition task into subtasks each solved by a simple expert. Jordan and Jacobs (1994) stacked gates into a tree, the hierarchical mixture of experts, and trained it with the EM algorithm. Two decades later, Eigen, Ranzato, and Sutskever (2013) put mixtures at several depths of one network, and Bengio, Léonard, and Courville (2013) framed the general goal of conditional computation. The modern form arrived with Shazeer et al. (2017), who placed a sparsely gated MoE layer with noisy top-k routing between LSTM layers, trained models with up to 137 billion parameters in the MoE layer and up to 131,072 experts, and solved the systems problems that made it practical. GShard (Lepikhin et al. 2020) and the Switch Transformer (Fedus et al. 2022) moved MoE into Transformers, and Mixtral and the DeepSeek models made it the default for open frontier models.

The 1991 paper contains an insight that every later MoE inherits, so it is worth one derivation. Suppose expert $i$ outputs $\mathbf{o}_i$ for a training case with target $\mathbf{d}$ , and the gate gives it weight $p_i$ . The obvious loss blends the experts and compares the blend with the target, $\lVert\mathbf{d} - \sum_i p_i\mathbf{o}_i\rVert^2$ . Jacobs et al. point out that this makes the experts cooperate. Each expert's gradient depends on the residual left by all the others, so every expert tries to fix everyone's errors, and many experts end up used for every case.

Their fix is to make the experts compete. Treat the gate as choosing one expert at random with probability $p_i$ , and treat each expert's output as the mean of a unit Gaussian over targets. The loss is the negative log-likelihood of the target under that mixture,

\mathcal{L} = -\log \sum_{i} p_i\, e^{-\frac12 \lVert \mathbf{d} - \mathbf{o}_i\rVert^2}.

Differentiate with respect to one expert's output. Only the $i$ -th term depends on $\mathbf{o}_i$ , and $\partial e^{-\frac12\lVert\mathbf{d}-\mathbf{o}_i\rVert^2}/\partial\mathbf{o}_i = e^{-\frac12\lVert\mathbf{d}-\mathbf{o}_i\rVert^2}(\mathbf{d}-\mathbf{o}_i)$ , so

\frac{\partial\mathcal{L}}{\partial\mathbf{o}_i} = -\,h_i\,(\mathbf{d} - \mathbf{o}_i), \qquad h_i = \frac{p_i\, e^{-\frac12\lVert\mathbf{d}-\mathbf{o}_i\rVert^2}}{\sum_j p_j\, e^{-\frac12\lVert\mathbf{d}-\mathbf{o}_j\rVert^2}} .

In plain words, each expert is pulled toward the target, weighted by $h_i$ , the posterior probability that it was the expert responsible. The expert that already fits best gets the largest share of the update, so experts specialize. Compare the intermediate loss $\sum_i p_i\lVert\mathbf{d}-\mathbf{o}_i\rVert^2$ , whose gradient is $-2p_i(\mathbf{d}-\mathbf{o}_i)$ . When the gate starts uniform and every expert's error $\lVert\mathbf{d}-\mathbf{o}_i\rVert$ exceeds 1, the paper notes, that version adapts the best-fitting expert the slowest (its error is smallest), and the mixture likelihood adapts it the fastest. The cell shows it.

Python · cell 4

The best-fitting expert gets the smallest update under the first loss and nearly all of it under the second. Modern Transformer MoEs do not use this exact loss (they backpropagate the language-modeling loss through a weighted sum, closer to the cooperative form), and one consequence is that they need explicit help to keep experts from all doing the same thing or one expert doing everything. That help is §13.3.

Core

Responsibilities in a mixture of experts

Implement responsibilities(target, O, p) for a target of shape (m,), expert outputs O of shape (E, m), and gate probabilities p of shape (E,). Return the pair (h, grad) where h of shape (E,) holds the posterior responsibilities $h_i$ above and grad of shape (E, m) is $\partial\mathcal{L}/\partial\mathbf{O}$ for $\mathcal{L} = -\log\sum_i p_i e^{-\frac12\lVert\mathbf{d}-\mathbf{o}_i\rVert^2}$ . It must stay finite when every expert is far from the target (squared errors in the thousands).

import numpy as np

def responsibilities(target, O, p):
    w = p * np.exp(-0.5 * ((target - O) ** 2).sum(1))
    h = w / w.sum()
    return h, -2 * p[:, None] * (target - O)       # placeholder gradient of the wrong loss

Hidden tests, hints, and AI help, from $1.99 a month.

In PyTorch

A readable MoE layer in PyTorch loops over experts, not tokens. For each expert it finds the tokens that chose it, runs them as one batch, and adds the weighted result back with index_add_. This is the shape of the reference Mixtral implementation in Hugging Face Transformers. §13.2 replaces the loop's bookkeeping with a sort, and §13.4 replaces the loop itself with a grouped matrix multiply.

🔥 In PyTorchfor a machine with PyTorch

Building the full-size layer on the meta device creates tensors with shapes but no storage, so you can count a 1.4-billion-parameter layer on a laptop. The count, $8\times176{,}160{,}768 + 32{,}768$ , is the experts plus the router of the hand count above.

	dense FFN, width $d_{\text{ff}}$	MoE, $E$ experts of width $d_e$ , top- $k$
parameters stored	$3\,d\,d_{\text{ff}}$	$3\,d\,d_e E + dE$
parameters used per token	$3\,d\,d_{\text{ff}}$	$3\,d\,d_e k + dE$
forward FLOPs per token	$6\,d\,d_{\text{ff}}$	$6\,d\,d_e k + 2dE$
memory to serve	small	large, all experts must be resident somewhere
extra machinery	none	router, load balancing, token dispatch

🎯 Interview questions on this section

What problem does a mixture of experts solve?

In a dense Transformer every parameter is used by every token, so the FLOPs per token are tied to the parameter count ( $2N$ forward, $6N$ to train). MoE decouples them. Each FFN is replaced by $E$ expert FFNs and a router that sends each token to $k \ll E$ of them, so stored parameters grow with $E$ while per-token compute grows with $k$ . You get the capacity (knowledge storage) of a large model at the training and inference FLOPs of a much smaller one, at the price of memory, communication, and routing complexity.

Mixtral 8x7B has 47B parameters and 13B active. Derive both, and explain why it is not 56B.

Per layer, attention with GQA is $2\cdot4096^2 + 2\cdot4096\cdot1024 = 41.9$ M, one SwiGLU expert is $3\cdot4096\cdot14336 = 176.2$ M, the router is $4096\cdot8$ , and two norms add $8192$ . With eight experts that is 1.451B per layer and 46.44B over 32 layers, plus 262M for the untied embeddings, so 46.70B. With two experts per token a layer uses 394.3M, so 12.88B including embeddings. It is not $8\times7.24 = 58$ B because only the FFNs (5.64B) are replicated, and the 1.60B of attention, norms, and embeddings is shared.

How do you estimate the training compute of an MoE model?

Use $C \approx 6N_{\text{active}}D$ plus the attention term, because unchosen experts do no work. For DeepSeek-V3, $6\times37\text{B}\times14.8\text{T} \approx 3.3\times10^{24}$ FLOPs, about eighteen times less than a dense 671B model. Add the router ( $2dE$ per token per layer, under 1% here). Then remember that FLOPs understate wall-clock cost, because MoE layers add all-to-all communication and run smaller, less efficient matrix products, so MFU is usually lower than for a dense model.

Is a mixture of experts an ensemble?

No. An ensemble runs several complete models and averages their predictions. An MoE layer routes each token separately in every layer, and the experts are slices of one FFN that share attention, embeddings, and the residual stream. Mathematically an MoE layer is a wide FFN of width $E d_e$ in which all but $k$ blocks of hidden units are switched off per token and the rest are scaled by gates. Mixtral has $\binom82^{32} \approx 2\times10^{46}$ possible paths, so there is no "expert 3 model" to speak of.

What did Shazeer et al. (2017) contribute beyond the 1991 mixture of experts?

Sparsity and scale. Earlier mixtures evaluated every expert, or used few experts. Shazeer et al. used noisy top-k gating so that only $k$ of up to tens of thousands of experts run per example, added losses to balance expert usage, and solved the shrinking batch problem (each expert sees only about $kB/E$ examples of a batch of $B$ ) by combining data parallelism for the dense parts with one shared copy of each expert fed by all devices. That made models with up to 137B parameters in the MoE layer trainable, with large gains in language modeling and translation at modest compute.

Why did Jacobs et al. (1991) replace the squared error of the blended output with a mixture likelihood?

With $\lVert\mathbf{d} - \sum_i p_i\mathbf{o}_i\rVert^2$ every expert's gradient depends on the residual left by all the others, so the experts cooperate and each case ends up using many of them. The negative log-likelihood $-\log\sum_i p_i e^{-\frac12\lVert\mathbf{d}-\mathbf{o}_i\rVert^2}$ gives expert $i$ the gradient $-h_i(\mathbf{d}-\mathbf{o}_i)$ with $h_i$ the posterior responsibility, so the expert that already fits best learns the most from the case. Experts compete and specialize, which was the point of the method.

Where does the MoE layer sit in a Transformer, and what stays dense?

It replaces the FFN sublayer, $\mathbf{h} = \mathbf{x} + \operatorname{MoE}(\operatorname{Norm}(\mathbf{x}))$ . Attention, norms, embeddings, and the output head stay dense and shared by all tokens. Many models use MoE in every layer (Mixtral), some in every other layer (GShard, GLaM), and DeepSeek-V3 keeps its first three layers dense. The router is a $d\times E$ linear map evaluated for every token.

🧭 What to remember

A dense model uses every parameter for every token, so compute per token is welded to parameter count. MoE replaces each FFN with $E$ experts and a router that picks $k$ of them, so storage grows with $E$ and compute with $k$ .
$\operatorname{MoE}(\mathbf{x}) = \sum_{i\in\mathcal{K}} g_i\operatorname{FFN}_i(\mathbf{x})$ with $\mathcal{K}$ the top- $k$ router logits and $g$ a softmax over them. Unchosen experts are never run.
Count total and active parameters separately. Mixtral has 46,702,792,704 and uses 12,879,925,248 per token. DeepSeek-V3 has 671,026,404,352 and uses about 37.6B.
Training compute is about $6N_{\text{active}}D$ , but memory scales with $N_{\text{total}}$ , and communication and small matmuls make MoE slower than its FLOPs suggest.
An MoE layer is a wide FFN with all but $k$ blocks of hidden units switched off per token, not an ensemble of models.
The idea runs from Jacobs et al. (1991), whose competitive loss made experts specialize, through Shazeer et al. (2017), who made it sparse and huge. Next, §13.2 opens up the router.

Jacobs, Jordan, Nowlan & Hinton (1991). Adaptive Mixtures of Local Experts. Five pages, and the cooperative versus competitive loss argument is still the clearest statement of why experts specialize.
Shazeer et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. Noisy top-k gating, balancing losses, the shrinking batch problem, and the first MoE at a hundred billion parameters.
Fedus, Zoph & Shazeer (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Top-1 routing in Transformers, and a careful account of the speed and quality trade-offs.
Jiang et al. (2024). Mixtral of Experts. A short paper with the full configuration used for the count in this section.
DeepSeek-AI (2024). DeepSeek-V3 Technical Report. The architecture, the 671B and 37B counts, and the training cost.
Fedus, Dean & Zoph (2022). A Review of Sparse Expert Models in Deep Learning. A survey of the whole field by people who built much of it.

Zero to Frontier is made for larger screens

Conditional computation

Parameters and FLOPs are welded together in a dense model

The MoE layer

A worked example small enough to check by hand

Total and active parameters

Mixtral 8x7B, counted exactly

DeepSeek-V3, counted exactly

Total and active parameters of an MoE model

FLOPs per token, and the 6N rule for MoE

Training compute of a dense model and its MoE counterpart

An MoE layer is a wide FFN with blocks switched off

Cut a wide FFN into experts

Where the idea came from

Responsibilities in a mixture of experts

In PyTorch

Read more

Watch next