Mixture of Experts (MoE)

Mixture of Experts is a conditional computation architecture that replaces dense feed-forward layers in a transformer with a set of parallel expert networks and a router. For each token, the router assigns weights to a small subset of experts (typically 2 out of 8, 16, or 64 total), and only those selected experts perform computation. The final output is a weighted sum of the chosen experts' outputs. Because most experts are idle for any given token, the model can have billions of parameters while activating only a fraction during inference—Mixtral 8x7B, for example, has 46.7B total parameters but only activates ~12.9B per token.

The practical consequence is a favorable trade-off between model capacity and compute. A dense 70B model requires 70B parameter operations per forward pass; an MoE model of equivalent quality might deliver similar performance while consuming the FLOP budget of a 13B model. This is why frontier labs (Google with Gemini, Mistral with Mixtral, and reportedly OpenAI with GPT-4) adopted MoE for their largest models. The challenges, however, are real: load balancing across experts requires auxiliary loss terms to prevent all tokens routing to the same few experts; serving MoE models requires all expert weights resident in VRAM even if only a subset activate, increasing memory pressure; and expert specialization is emergent rather than designed, making interpretability harder.

For developers building agent systems and AI products, MoE matters in two ways. First, open MoE models like Mixtral offer near-frontier quality at a fraction of the serving cost of equivalent dense models, making them compelling for high-throughput agentic workloads. Second, understanding MoE helps when evaluating model cards and benchmarks—a model's total parameter count and its *active* parameter count are fundamentally different numbers with different cost implications for inference infrastructure.