
Principal Components Regression and Partial Least Squares
Part of Stanford's Statistical Learning course, taught by Trevor Hastie, Robert Tibshirani, and Jonathan Taylor of Stanford's Department of Statistics. This session covers two methods for handling high dimensional predictor sets through dimension reduction rather than variable selection. Principal components regression builds new predictor directions that capture the greatest variance in the predictor matrix, then regresses the response on a small number of these components. Partial least squares takes a related approach but constructs its components using the response variable itself, seeking directions that are both high variance and correlated with the outcome. The lecture compares how each method trades off bias and variance against ordinary least squares and ridge regression, and discusses when reducing to a handful of components can outperform fitting all original predictors. Equations and plots appear on slides as the instructors walk through the derivations and connect the methods back to the course's running theme of prediction accuracy versus interpretability.