
Behind the Scenes of LLM Pre-training: StarCoder Use Case
Loubna Ben Allal of Hugging Face walks through what it actually takes to pretrain a large language model, drawing on her work with the BigCode Project. Part of Stanford's CS25: Transformers United seminar series, the talk covers scaling laws, data curation, and the practical choices that shape a training run before it ever produces useful output. She uses StarCoder, the code-generation model she helped build and co-authored alongside The Stack dataset, as a worked example of how training a model for code differs from training a general-purpose LLM. The second half turns to data governance and evaluation, topics she argues get far less attention than the pretraining stage itself despite being just as consequential for how these models end up behaving. The talk is aimed at people who already know what an LLM is and want the engineering and data decisions behind one specific, real system.