LECTURES A GRATIS GLOBAL SERVICE
⌕ SEARCH GRATIS GLOBAL ↗
LECTURES
Scaling Rules for Optimization
SOURCE: YOUTUBE · NO TRACKING UNTIL YOU PRESS PLAY · TROUBLE PLAYING? WATCH AT THE SOURCE ↗

Scaling Rules for Optimization

81 MIN · EN · STATUS: [ STREAMING ]
RATE THIS
MIT · Deep Learning · LECTURE 7

Jeremy Bernstein lectures on hyperparameter transfer in deep learning, part of MIT's 6.7960 Deep Learning course, Fall 2024. He approaches neural network training from a spectral perspective, analyzing how weight matrices and gradients behave under different norms. The lecture develops the idea of feature learning as a lens for understanding why certain hyperparameters, like learning rate, need to change predictably as network width and depth grow. Bernstein presents scaling rules that let practitioners tune a small model and transfer those settings to a much larger one without re-running expensive searches. The talk moves through the mathematical machinery behind these rules, connecting abstract spectral norms to concrete practical advice for training large networks efficiently. It runs 81 minutes and assumes familiarity with the course's earlier material on optimization and network architecture.

More from this course

12 LECTURES
Introduction to Deep Learning

Introduction to Deep Learning

MIT · 61 MIN
How to Train a Neural Net

How to Train a Neural Net

MIT · 80 MIN
Approximation Theory

Approximation Theory

MIT · 83 MIN
Architectures: Grids

Architectures: Grids

MIT · 84 MIN
Architectures: Graphs

Architectures: Graphs

MIT · 81 MIN
Generalization Theory

Generalization Theory

MIT · 81 MIN
Architectures: Transformers

Architectures: Transformers

MIT · 75 MIN
Hacker's Guide to Deep Learning

Hacker's Guide to Deep Learning

MIT · 76 MIN
Architectures: Memory

Architectures: Memory

MIT · 73 MIN
Lec 11: Representation Learning: Reconstruction-Based

Lec 11: Representation Learning: Reconstruction-Based

MIT · 81 MIN
Representation Learning: Similarity-Based

Representation Learning: Similarity-Based

MIT · 76 MIN
Lecture 13: Representation Learning Theory

Lecture 13: Representation Learning Theory

MIT · 75 MIN