LECTURES A GRATIS GLOBAL SERVICE
⌕ SEARCH GRATIS GLOBAL ↗
LECTURES
Attention Approximates Sparse Distributed Memory
SOURCE: YOUTUBE · NO TRACKING UNTIL YOU PRESS PLAY · TROUBLE PLAYING? WATCH AT THE SOURCE ↗

Attention Approximates Sparse Distributed Memory

82 MIN · EN · STATUS: [ STREAMING ]
RATE THIS
STANFORD

Trenton Bricken, a PhD student at Harvard, and Will Dorrell, a PhD student at University College London's Gatsby Unit, present a guest lecture in Stanford's CS25: Transformers United series. They argue that the attention mechanism at the core of transformer models can be understood as an approximation of Sparse Distributed Memory, a decades-old theory of how neural systems store and retrieve information. The talk works through the mathematical correspondence between the two models, drawing on neuroscience literature to explain why attention's behavior resembles biological memory retrieval. Bricken and Dorrell walk through the architecture of transformers piece by piece, connecting each component to its proposed neural analogue, and discuss what this link might suggest about building more brain-like AI systems. The session runs 82 minutes and assumes familiarity with transformer basics from earlier lectures in the series.

At a glance

Lecture facts

Runtime compared with the other 387 Computer Science lectures
Runtime1 h 22 m
Compared with Computer ScienceLonger than 82%
Source channelStanford Online (YouTube)