Mixture of Experts

It is January 2017. The Transformer does not exist yet — sequence models are still stacks of LSTM layers, the architecture Chapter 3 of the language-models book covers — and a team at Google has already run into the problem this chapter is about: making a dense neural network bigger multiplies the compute spent on every single token, whether or not that token needed the extra capacity. A paper published that month proposes a layer that is, in its own words, outrageously large in parameter count while staying nearly constant in per-token compute. It will take four more years to reach the Transformer, and eight more to reach a 671-billion-parameter model trained for $5.576 million.

Where We Were

A dense feed-forward or LSTM layer touches every one of its parameters for every input it processes. Doubling a dense layer’s width doubles both its parameter count and the multiply-accumulate operations needed to run it — capacity and compute move in lockstep, by construction. Chapters 8 and 9 cover how data, tensor, and pipeline parallelism let a team spread that compute across more accelerators, but none of those techniques break the lockstep: they let a bigger dense model fit and run faster across a cluster, without changing the fact that every added parameter costs every token something.

Conditional computation — the idea that different inputs might route through different parts of a network, so most of the network can stay idle for most inputs — had been proposed in various forms through the early-to-mid 2010s, but no version of it scaled past small experiments. The obstacle was not the idea but the engineering: choosing a subset of a network to activate is a discrete decision, discrete decisions do not have gradients, and any routing mechanism has to keep the units it activates busy with enough work to justify the communication cost of getting inputs to them across accelerators, or the exercise saves no wall-clock time even as it saves arithmetic.

The routing idea arrives five months before the Transformer, then sits outside the mainstream for four years before it matters again. PNG ↓

The Breakthrough

Shazeer et al. at Google proposed the Sparsely-Gated Mixture-of-Experts layer in January 2017 (Shazeer et al., 2017), inserting it between stacked LSTM layers rather than replacing a single feed-forward block. The layer holds up to thousands of identically-shaped feed-forward sub-networks — the experts — plus a small trainable router that looks at each token’s representation and picks a handful of experts to send it to, ignoring the rest entirely. Because the router’s choice is discrete, gradients cannot flow through it directly by the usual chain rule; the paper’s routing function adds tunable Gaussian noise to the router’s raw scores before selecting the top few, which both makes the discrete choice differentiable in expectation and, as a side effect, spreads tokens more evenly across experts than a noiseless top-k choice would. Trained as part of language models with up to 137 billion parameters in the MoE layer alone, and using up to 131,072 experts in the largest configuration, the resulting models reported a perplexity of 28.0 on a 1-billion-word language modeling benchmark against a prior state of the art of 34.7, and a BLEU score of 40.56 on WMT’14 English-to-French translation against a 39.22 baseline — the paper’s own framing was “greater than 1000x improvements in model capacity with only minor losses in computational efficiency.”

The idea then sat mostly outside the mainstream for four years. The field’s attention shifted to the Transformer, published five months later in June 2017 — language models, Chapter 5 covers it in full — and reproducing Shazeer et al.’s routing and load-balancing machinery inside a Transformer’s feed-forward blocks, at a scale that justified the added engineering, took until Fedus, Zoph, and Shazeer published the Switch Transformer in January 2021 (Fedus et al., 2021). Switch Transformers made one deliberate simplification: instead of routing each token to several experts, as the original layer and most follow-up work did, it routes each token to exactly one — switch routing. Sending a token to a single expert rather than several cuts the communication volume between accelerators roughly in proportion to how many experts each token would otherwise have visited, and the paper reports that this simplification does not cost quality: a 64-expert Switch-Base model reached a T5-Base model’s quality roughly 7.5 times faster in training steps at matched compute, and the largest configuration, Switch-C, scaled to 2,048 experts and roughly 1.571 trillion parameters — the first published trillion-parameter model — training in bfloat16 without the instability the field had come to expect from models at that scale.

Reproducing that recipe at the scale that mattered next required an openly available model, and Mistral AI supplied one with Mixtral 8x7B in January 2024 (Jiang et al., 2024), continuing the open-weights lineage covered in language models, Chapter 14. Mixtral widens the router back out to top-2 — each token visits two of eight experts per layer — landing between Shazeer et al.’s top-k and Switch’s top-1. The released model holds 47 billion total parameters but uses only 13 billion of them for any given token, and Mistral AI reported it matching or exceeding a dense 70-billion-parameter Llama 2 model and GPT-3.5 on most benchmarks, including an MMLU score of 70.6% against Llama 2 70B’s 69.9% and GPT-3.5’s reported 70.0%, while running inference at roughly the cost of a 13-billion-parameter dense model. It shipped under the Apache 2.0 license, making it the first widely-used sparse MoE model that any team could inspect, fine-tune, or self-host.

DeepSeek-AI closed out this lineage in December 2024 with DeepSeek-V3, a 671-billion-parameter model activating 37 billion parameters per token (DeepSeek-AI, 2024). Its MoE layers use 256 routed experts plus one always-active shared expert, with each token routed to the shared expert and the top 8 of the 256 routed experts — more granular than Mixtral’s top-2-of-8, and without Switch’s single-expert bottleneck. DeepSeek-V3’s specific contribution to routing is a fix for a longstanding side effect of the auxiliary load-balancing losses every prior model in this lineage relied on: forcing traffic to be even across experts, via an extra loss term added to the training objective, pulls the model’s parameters away from whatever direction the main training loss alone would have chosen, at every single step. DeepSeek-V3 replaces that loss term with a per-expert bias that only affects which experts get picked, not the training objective at all — detailed under The Mathematics below — reporting a full training run — pretraining plus context extension plus post-training — of 2.788 million H800 GPU-hours, a total training cost of $5.576 million, and no irrecoverable loss spikes across the entire run. Trained in FP8 mixed precision (covered in Chapter 7), the resulting model reached an MMLU score of 87.1 against LLaMA 3.1 405B’s 84.4, and an MMLU-Pro score of 64.4 against 52.8 — a dense 405-billion-parameter model beaten on both by a sparse model activating well under a tenth of its total parameters per token.

Only a handful of experts ever run for a given token, so total compute per token stays fixed no matter how many thousand experts the layer holds. PNG ↓

Mixtral scores all eight experts and keeps only two; the other six were never idle, they were simply outvoted. (animates as you scroll to it) PNG ↓

The router's fan-out widens, narrows to one, then widens again; each generation picks a different point on the same top-k spectrum. PNG ↓

The Mathematics

Shazeer et al.’s router scores every expert for a given input, adds tunable noise to break ties and spread load, and keeps only the top $k$ scores before turning them into weights:

G(x) = \mathrm{Softmax}\big(\mathrm{KeepTopK}(H(x), k)\big), \qquad H(x)_i = (x W_g)_i + \mathcal{N}(0,1) \cdot \mathrm{Softplus}\big((x W_{\mathrm{noise}})_i\big)

$x$ is the token’s input representation. $W_g$ and $W_{\mathrm{noise}}$ are learned weight matrices that turn $x$ into, respectively, a raw preference score and a per-expert noise magnitude for every expert. $H(x)_i$ is expert $i$ ‘s noisy score — its clean preference score plus Gaussian noise scaled by a learned, input-dependent amount. $\mathrm{KeepTopK}$ zeroes out every score except the $k$ largest, and the surrounding softmax turns the surviving scores into weights that sum to one. $G(x)$ is the final gate vector: nonzero only for the $k$ chosen experts. What this buys you: only the $k$ selected experts ever run for a given token, so total compute per token stays fixed regardless of how many thousands of experts the layer holds, while the learned noise term makes an otherwise non-differentiable top- $k$ choice trainable and discourages the router from always picking the same few experts.

Symbol	Meaning
$x$	token’s input representation
$W_g$ , $W_{\mathrm{noise}}$	learned weights producing preference scores and noise scale
$H(x)_i$	expert $i$ ‘s noisy routing score
$k$	number of experts kept per token
$\mathrm{KeepTopK}$	zeroes every score outside the top $k$
$G(x)$	final gate vector, nonzero only for chosen experts

Switch Transformers simplify $k$ to 1 and route with a plain softmax over a linear projection $h(x)$ , sending each token’s full weight to a single expert:

\begin{aligned} p_i(x) &= \frac{\exp\big(h(x)_i\big)}{\sum_{j=1}^{N}\exp\big(h(x)_j\big)} \\ y &= p_{i^*}(x)\, E_{i^*}(x), \qquad i^* = \arg\max_i\, p_i(x) \end{aligned}

Symbol	Meaning
$N$	total number of experts in the layer
$h(x)$	linear projection of the input used to score experts
$p_i(x)$	router’s softmax probability for expert $i$
$i^*$	index of the single highest-probability expert
$y$	layer output
$E_{i^*}$	the selected expert’s feed-forward block

Sending every token to whichever expert scores highest, with nothing else, tends to concentrate traffic on a few popular experts and starve the rest — the exact failure mode the training data punishes as wasted capacity, not the router. Switch Transformers, like Shazeer et al. before them, counteract this with an auxiliary loss added to the training objective:

\mathcal{L}_{\mathrm{aux}} = \alpha\, N \sum_{i=1}^{N} f_i\, P_i, \qquad f_i = \frac{1}{T}\sum_{x \in \mathcal{B}} \mathbb{1}\{i^*(x) = i\}, \qquad P_i = \frac{1}{T}\sum_{x \in \mathcal{B}} p_i(x)

$\mathcal{B}$ is a training batch of $T$ tokens. $f_i$ is the actual fraction of tokens in the batch that got routed to expert $i$ , and $P_i$ is the fraction of total router probability mass expert $i$ received — a softer, differentiable stand-in for the same quantity, since $f_i$ alone has no useful gradient. $\alpha$ is a small fixed weight (Switch Transformers use $10^{-2}$ ) controlling how strongly this loss competes with the main training objective. What this buys you: $\sum_i f_i P_i$ is minimized when both quantities are spread evenly across all $N$ experts, so adding $\mathcal{L}_{\mathrm{aux}}$ to the loss directly penalizes a router that dumps most tokens on a handful of experts — at the cost of nudging every training step slightly away from whatever direction the main loss alone would have picked.

Symbol	Meaning
$\mathcal{B}$ , $T$	training batch and its token count
$f_i$	fraction of tokens actually routed to expert $i$
$P_i$	fraction of router probability mass assigned to expert $i$
$\alpha$	fixed coefficient weighting the auxiliary loss
$N$	total number of experts

Mixtral widens the router to the top 2 of $N$ experts and combines their outputs directly, with no separate load-balancing term appearing in its published objective:

G(x) = \mathrm{Softmax}\big(\mathrm{Top2}(x W_g)\big), \qquad y = \sum_{i=1}^{N} G(x)_i \, \mathrm{SwiGLU}_i(x)

Symbol	Meaning
$N$	number of experts per layer, 8 in Mixtral
$W_g$	router’s learned weight matrix
$\mathrm{Top2}$	keeps the two highest scores, discards the rest
$\mathrm{SwiGLU}_i$	expert $i$ ‘s feed-forward block
$y$	layer output, combining the two chosen experts

DeepSeek-V3 keeps a Switch-style top- $k$ selection but decides membership using a score the training loss never touches directly:

i \in \mathrm{Top}k_{K_r}\big(\{\, s_{j,t} + b_j \;:\; 1 \le j \le N_r \,\}\big)

$N_r$ is the number of routed experts, 256 in DeepSeek-V3, and $K_r$ is how many are chosen per token, 8. $s_{j,t}$ is expert $j$ ‘s ordinary routing affinity score for token $t$ , computed the same way a Switch- or Mixtral-style router would. $b_j$ is a separate bias term, one per expert, that affects only which experts make the $K_r$ selected — the gate weight the model actually multiplies by still uses $s_{j,t}$ alone, with $b_j$ excluded. After every training batch, DeepSeek-V3 nudges each $b_j$ down by a small fixed step if expert $j$ was overloaded that batch and up if it was underloaded, with the step size annealed to zero partway through training. What this buys you: load balancing without any term added to the training loss at all — the same evenness the auxiliary-loss approach bought at the cost of distorting every gradient step is instead bought by a bias that only ever touches expert selection, leaving the model’s actual optimization objective alone.

Symbol	Meaning
$N_r$ , $K_r$	number of routed experts and how many are selected per token
$s_{j,t}$	expert $j$ ‘s routing affinity score for token $t$
$b_j$	per-expert bias, adjusted after each batch, excluded from the gate weight

What It Enabled

	Dense scaling (Chapters 8–9)	Sparse MoE (this chapter)
Relationship between size and per-token compute	proportional — doubling parameters roughly doubles FLOPs per token	decoupled — total parameters can grow far faster than active parameters per token
Representative model	any model that fits the cluster, run in full for every token	Switch-C, 1.571T total parameters, 2,048 experts (2021)
Frontier example	dense 405B-parameter LLaMA 3.1	DeepSeek-V3, 671B total / 37B active parameters (2024)
Reported result at frontier	MMLU 84.4, MMLU-Pro 52.8 (LLaMA 3.1 405B)	MMLU 87.1, MMLU-Pro 64.4 (DeepSeek-V3)

Model	Year	Routing	Total params	Active params/token	Load balancing
Sparsely-Gated MoE	2017	noisy top- $k$	up to 137B in MoE layer	varies with $k$	importance + load losses
Switch Transformer (Switch-C)	2021	top-1	~1.571T	fraction of one expert per token	auxiliary loss, $\alpha=10^{-2}$
Mixtral 8x7B	2024	top-2 of 8	47B	13B	none published
DeepSeek-V3	2024	top-8 of 256 + 1 shared	671B	37B	auxiliary-loss-free bias

Decoupling capacity from per-token compute changed what “bigger model” meant for a training budget: total parameters stopped being a reliable predictor of either training or inference cost, and active parameters per token became the number that mapped to GPU-hours. That shift interacts directly with how a sparse model gets served rather than trained — a request that only touches 37 billion of DeepSeek-V3’s 671 billion parameters still needs every one of those 671 billion in accelerator memory somewhere in the cluster, since which experts get used shifts from token to token and batch to batch, a scheduling and memory problem Chapter 17 covers. It also raised the value of getting inter-chip communication right, since a router that spreads tokens across experts living on different accelerators needs an all-to-all exchange every layer, tying MoE more tightly to the interconnect story in Chapter 20. And a 671-billion-parameter model that costs $5.576 million to pretrain, rather than the tens of millions a dense model of comparable benchmark performance would have cost, changed the economics of what came after it: DeepSeek-V3 became the base model for DeepSeek-R1, a reasoning model trained with reinforcement learning and covered in language models, Chapter 20, and cheaper frontier-scale pretraining runs are a direct input to the test-time-compute economics covered in Chapter 19.

A sparse frontier model's routing choices ripple into reasoning training, serving, hardware, and the economics of thinking longer, all at once. PNG ↓

Key Papers

Paper	Authors	Year	Link	Why it matters
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer	Shazeer et al.	2017	arXiv:1701.06538	Introduced noisy top-k routing and load-balancing losses, decoupling model capacity from per-token compute for the first time at scale
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	Fedus et al.	2021	arXiv:2101.03961	Simplified routing to top-1, cutting communication cost and reaching the first published trillion-parameter model
Mixtral of Experts	Jiang et al.	2024	arXiv:2401.04088	First widely-used open-weights sparse MoE, matching a 70B dense model’s benchmarks at 13B active parameters
DeepSeek-V3 Technical Report	DeepSeek-AI	2024	arXiv:2412.19437	Replaced the auxiliary load-balancing loss with a bias-only mechanism, reaching 671B total parameters for $5.576M in training cost

PM Lens

The number I keep coming back to from this chapter is $5.576 million — DeepSeek-V3’s full reported pretraining cost for a model that beat a 405-billion-parameter dense model on MMLU and MMLU-Pro. That number only makes sense once you separate two things product conversations routinely collapse into one: how much a model “knows” and how much it costs to run. A 671-billion-parameter MoE model and a 70-billion-parameter dense model can serve a request at similar latency and cost, because the MoE model only wakes up a fraction of itself per token; the headline parameter count tells you almost nothing about the bill. When a vendor pitches a bigger model, or a team wants to train one, my first question now is “active parameters per token, not total,” because that number, not the marketing figure, is what shows up in the inference invoice. Mixtral’s Apache 2.0 license mattered for the same reason — it let smaller teams get a real answer to that question themselves, instead of taking a lab’s word for it.

Search