
Scaling Rules for Optimization
Jeremy Bernstein lectures on hyperparameter transfer in deep learning, part of MIT's 6.7960 Deep Learning course, Fall 2024. He approaches neural network training from a spectral perspective, analyzing how weight matrices and gradients behave under different norms. The lecture develops the idea of feature learning as a lens for understanding why certain hyperparameters, like learning rate, need to change predictably as network width and depth grow. Bernstein presents scaling rules that let practitioners tune a small model and transfer those settings to a much larger one without re-running expensive searches. The talk moves through the mathematical machinery behind these rules, connecting abstract spectral norms to concrete practical advice for training large networks efficiently. It runs 81 minutes and assumes familiarity with the course's earlier material on optimization and network architecture.