LECTURES A GRATIS GLOBAL SERVICE
⌕ SEARCH GRATIS GLOBAL ↗
LECTURES
Behind the Scenes of LLM Pre-training: StarCoder Use Case
SOURCE: YOUTUBE · NO TRACKING UNTIL YOU PRESS PLAY · TROUBLE PLAYING? WATCH AT THE SOURCE ↗

Behind the Scenes of LLM Pre-training: StarCoder Use Case

62 MIN · EN · STATUS: [ STREAMING ]
RATE THIS
STANFORD

Loubna Ben Allal of Hugging Face walks through what it actually takes to pretrain a large language model, drawing on her work with the BigCode Project. Part of Stanford's CS25: Transformers United seminar series, the talk covers scaling laws, data curation, and the practical choices that shape a training run before it ever produces useful output. She uses StarCoder, the code-generation model she helped build and co-authored alongside The Stack dataset, as a worked example of how training a model for code differs from training a general-purpose LLM. The second half turns to data governance and evaluation, topics she argues get far less attention than the pretraining stage itself despite being just as consequential for how these models end up behaving. The talk is aimed at people who already know what an LLM is and want the engineering and data decisions behind one specific, real system.

At a glance

Lecture facts

Runtime compared with the other 387 Computer Science lectures
Runtime1 h 2 m
Compared with Computer ScienceShorter than 61%
Source channelStanford Online (YouTube)