LECTURES A GRATIS GLOBAL SERVICE
⌕ SEARCH GRATIS GLOBAL ↗
LECTURES
Demystifying Mixtral of Experts
SOURCE: YOUTUBE · NO TRACKING UNTIL YOU PRESS PLAY · TROUBLE PLAYING? WATCH AT THE SOURCE ↗

Demystifying Mixtral of Experts

65 MIN · EN · STATUS: [ STREAMING ]
RATE THIS
STANFORD

Albert Jiang, AI scientist at Mistral AI and PhD candidate at Cambridge, explains the Mixtral 8x7B sparse mixture of experts language model in this Stanford CS25 Transformers United talk. He walks through how Mixtral shares its architecture with Mistral 7B but replaces each feedforward layer with eight expert blocks, with a router network selecting two experts per token at each layer. Jiang breaks down why this gives the model access to 47 billion parameters while only activating 13 billion during inference, and he presents his analysis of the routing decisions the model actually makes, looking at which experts get chosen and whether patterns emerge across tokens and layers. The talk covers architectural tradeoffs behind sparse mixture of experts designs and what the routing data reveals about how the model organizes its computation.

At a glance

Lecture facts

Runtime compared with the other 734 Mathematics lectures
Runtime1 h 5 m
Compared with MathematicsShorter than 69%
Source channelStanford Online (YouTube)