Mixture of experts: how a huge model uses only a small part of itself per token

A mixture-of-experts (MoE) model is a neural network that contains many parallel sub-networks, called experts, but sends each piece of text through only a few of them. A small component called the router decides which ones. The model can therefore store far more knowledge, measured in parameters, than it spends compute on for any single token.

This is the idea behind figures like “120B parameters, 12B active”. The first number is the size of the whole model. The second is how much of it actually runs for each token.

Splitting the feedforward layer into many specialists

In a standard transformer, every layer has an attention block and a feedforward block, and every token passes through all of them. The feedforward block holds most of the parameters. In a dense model, all of those parameters work on every token.

A mixture-of-experts layer replaces that single feedforward block with several parallel ones, the experts. A small router network looks at each token and scores the experts. Only the top-scoring few run, and their outputs are blended using the router’s scores. The remaining experts sit idle for that token.

An analogy: a hospital with dozens of specialists on staff. Each patient is seen by the two or three doctors the triage nurse picks, not by everyone. The hospital’s total expertise is large, but the cost of treating one patient stays small. The nurse is the router, and the quality of triage matters as much as the quality of the doctors.

Two details are easy to miss. First, “specialist” is a loose word. Experts are not assigned topics such as law or chemistry by design. The router and experts are trained together, and whatever division of labor emerges is learned, not specified. Second, routing happens at every MoE layer and for every token, so one sentence can use many different combinations of experts as it moves through the network.

From a 2017 Google paper to trillion-parameter experiments

The modern form comes from the 2017 paper “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” by Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton and Jeff Dean. Their argument was that a network’s capacity is limited by its parameter count, and that conditional computation, where only parts of the network activate for each input, could raise capacity without a matching rise in compute. Their layer held up to thousands of feedforward sub-networks, with a trainable gating network choosing a sparse subset. They built models with up to 137 billion parameters, placed between stacked LSTM layers (the dominant architecture before transformers), and reported capacity gains of more than 1000x with only minor losses in computational efficiency. They tested it on language modeling and machine translation.

In 2021, William Fedus, Barret Zoph and Noam Shazeer published the Switch Transformer. They noted that MoE adoption had been held back by complexity, communication costs and training instability, and they simplified the routing algorithm and proposed training techniques to tame the instability. They reported up to 7x faster pre-training than dense T5-Base and T5-Large models using the same compute, a 4x speedup over T5-XXL for a trillion-parameter model, and that large sparse models could be trained in the lower-precision bfloat16 format.

The idea then moved into widely used open models. In January 2024, Mistral AI released Mixtral 8x7B (paper by Albert Q. Jiang and colleagues). It shares the architecture of Mistral 7B, but each layer has eight feedforward experts, and the router picks two per token. It has about 47B total parameters and about 13B active during inference, and the authors report that it matches or beats Llama 2 70B and GPT-3.5 on the benchmarks they evaluated. Both the base and instruct versions were released under the Apache 2.0 license. The total is below a naive 8 x 7B because only the feedforward blocks are duplicated into experts; the rest of each layer is shared.

Why 120B of storage can cost like 12B of compute

Compute per token scales with the active parameters, not the total. A model with 120B parameters of which 12B fire per token needs roughly the arithmetic of a 12B dense model for each step, while drawing on a much larger pool of learned knowledge. That gap is the whole appeal: more capacity for the same inference compute, or the same capacity for less.

The saving applies to compute, not to memory. All experts must be loaded and ready, because the next token may need any of them. A 120B MoE model needs hardware that can hold 120B parameters, even though each token only touches a tenth of them. The practical gain is speed and cost per token, not a smaller footprint.

Where mixture of experts shows up in current models

The pattern is now common in large language models. Nvidia’s Nemotron 3.5 Super VL, which AWS added to SageMaker JumpStart, is one example: it has 120B parameters, about 12B active per token, and a language backbone that our report on the AWS release describes as a hybrid Mixture-of-Experts design paired with a vision encoder. The same report notes that the active-parameter count helps explain the speed figures Nvidia and third-party testers cite, which came from a pre-release endpoint and may differ on production hardware. Nvidia’s claim of finishing tasks about five times faster than the nearest competitor in its class is a company claim that has not been independently verified.

MoE is also combined with other efficiency ideas. In Nemotron’s case, the backbone is a hybrid design that, per the report, aims to keep long-context cost down, which matters when a 1M-token window has to hold hours of video frames or many document pages. MoE reduces the cost of each token; the hybrid layers target the cost of long sequences. They address different bottlenecks and stack well.

Routing, balance and deployment trade-offs

The Switch Transformer authors named the core difficulties directly: complexity, communication costs and training instability. Each is still a real constraint.

• Routing quality. If the router sends most tokens to a few favorite experts, the others receive little training and the model behaves like a smaller one. Training has to encourage a balanced load.
• Communication. When experts are spread across many GPUs, tokens must travel to the devices holding their chosen experts and back. This adds network traffic that dense models do not have.
• Instability. Sparse models can be harder to train than dense ones, which is why the Switch paper devotes effort to stabilizing techniques.
• Memory. As noted above, total parameters still have to fit in memory, so MoE models need large accelerators or several of them even when their active size is small.
• Fine-tuning. Adapting a model with many experts is more involved than adapting a dense one, since the router and the experts both have to behave well on the new data. This is one reason providers publish separate weights for customization, as AWS does with its BF16 reference weights.

Comparisons also need care. “Active parameters” is a useful label for compute, but it does not translate directly into quality. A 12B-active MoE model is not guaranteed to match a 120B dense one; results depend on training data, routing and the benchmark, and should be checked on the task that matters.

What to watch as sparse models spread

Three things are worth following. The first is how far the active fraction keeps shrinking: Mixtral used two of eight experts per layer, while newer models cited in the news use roughly a tenth of their parameters per token. The second is how MoE combines with long-context and multimodal designs, since video and large documents raise the cost of every token. The third is serving: cloud catalogs now offer MoE models as ready-made deployments, and the real-world numbers to compare are latency, throughput and price per token on actual hardware, not parameter counts alone. For the Nemotron release on AWS, the announcement gave neither pricing nor a list of supported Regions.