
RAG Explained: From BM25 to Agentic RAG
Stanford professor Diyi Yang teaches the third lecture of CS329Z: Engineering AI Agents, walking through retrieval-augmented generation from fundamentals to current research. She covers why grounding model answers in retrieved evidence matters, the shift from retriever-reader QA systems to full RAG pipelines, and indexing choices like chunking and contextual retrieval. The lecture compares BM25 against dense retrieval methods such as DPR, explains vector search with HNSW, and works through hybrid search and reranking with bi-encoders, cross-encoders, and ColBERT. Later sections move into hierarchical and graph-based approaches, RAPTOR and GraphRAG, before closing on agentic RAG methods including Self-RAG, CRAG, and Search-R1, plus a discussion of common failure modes. Two built-in Q&A segments address whether frontier models still need retrieval at all and how errors compound in graph-based systems. The talk assumes graduate-level familiarity with machine learning and gives a technical, pipeline-by-pipeline account of how modern retrieval-augmented systems are built.