LECTURES A GRATIS GLOBAL SERVICE
⌕ SEARCH GRATIS GLOBAL ↗
LECTURES
Language Modeling from Scratch: Overview and Tokenization
SOURCE: YOUTUBE · NO TRACKING UNTIL YOU PRESS PLAY · TROUBLE PLAYING? WATCH AT THE SOURCE ↗

Language Modeling from Scratch: Overview and Tokenization

79 MIN · EN · STATUS: [ STREAMING ]
RATE THIS
STANFORD

Stanford's CS336 opens with Percy Liang and Tatsunori Hashimoto laying out what it means to build a language model from the ground up rather than fine-tune an existing one. The lecture covers the course's philosophy, why understanding every layer of the stack, from data to systems to scaling laws, matters for researchers who will otherwise treat large models as black boxes. It surveys the current landscape of open and closed models, the growing gap in transparency between them, and then turns to tokenization, walking through byte pair encoding and the tradeoffs involved in turning raw text into the discrete units a model trains on. Liang and Hashimoto frame tokenization as the first of many unglamorous but consequential design choices the course will work through in detail. It sets expectations for a class built around implementation, not just theory, with assignments that require writing the components discussed rather than only reading about them.

At a glance

Lecture facts

Runtime compared with the other 387 Computer Science lectures
Runtime1 h 19 m
Compared with Computer ScienceLonger than 71%
Source channelStanford Online (YouTube)