
Attention Approximates Sparse Distributed Memory
Trenton Bricken, a PhD student at Harvard, and Will Dorrell, a PhD student at University College London's Gatsby Unit, present a guest lecture in Stanford's CS25: Transformers United series. They argue that the attention mechanism at the core of transformer models can be understood as an approximation of Sparse Distributed Memory, a decades-old theory of how neural systems store and retrieve information. The talk works through the mathematical correspondence between the two models, drawing on neuroscience literature to explain why attention's behavior resembles biological memory retrieval. Bricken and Dorrell walk through the architecture of transformers piece by piece, connecting each component to its proposed neural analogue, and discuss what this link might suggest about building more brain-like AI systems. The session runs 82 minutes and assumes familiarity with transformer basics from earlier lectures in the series.