
Demystifying Mixtral of Experts
Albert Jiang, AI scientist at Mistral AI and PhD candidate at Cambridge, explains the Mixtral 8x7B sparse mixture of experts language model in this Stanford CS25 Transformers United talk. He walks through how Mixtral shares its architecture with Mistral 7B but replaces each feedforward layer with eight expert blocks, with a router network selecting two experts per token at each layer. Jiang breaks down why this gives the model access to 47 billion parameters while only activating 13 billion during inference, and he presents his analysis of the routing decisions the model actually makes, looking at which experts get chosen and whether patterns emerge across tokens and layers. The talk covers architectural tradeoffs behind sparse mixture of experts designs and what the routing data reveals about how the model organizes its computation.