
Lecture 16: Data Compression and Shannon's Noiseless Coding Theorem
MIT professor Ankur Moitra teaches this lecture from 18.200, Principles of Discrete Applied Mathematics, covering the mathematics of data compression. He opens with the history of the problem, then defines a first-order information source and what it means to compress one. From there he builds up the definition of entropy as a measure of information content, and uses it to state and prove Shannon's noiseless coding theorem, which establishes the optimal compression ratio achievable for a first-order source. The sixty-nine minute session is blackboard-style, with Moitra working through definitions and the proof step by step rather than relying on slides. It assumes prior exposure to probability and discrete math from earlier in the course, and serves as the rigorous foundation for why compression algorithms cannot beat the entropy bound.