
Decision Making Under Uncertainty: Policy Gradient Estimation and Optimization
Joshua Ott of Stanford University lectures on policy gradient methods for reinforcement learning, part of the AA228/CS238 Decision Making Under Uncertainty course. The session covers how to estimate gradients of expected reward with respect to policy parameters when the underlying dynamics are stochastic or unknown, and how to use those estimates to optimize a policy directly rather than through value functions. Ott works through the mathematics of gradient estimation, including the likelihood ratio and finite-difference approaches, and discusses the practical tradeoffs of variance and bias in these estimators. The lecture sits within a broader course on computational approaches to decision making, including Bayesian networks, dynamic programming, and partially observable Markov decision processes, with applications drawn from autonomous vehicles, air traffic control, and robotic exploration. Runs 81 minutes as a standalone graduate-level session.