
From Large Language Models to Large Multimodal Models
Ming Ding, a research scientist at Zhipu AI and Tsinghua PhD, delivers this Stanford CS25 Transformers United guest lecture on the move from text-only language models toward multimodal systems. He covers the basics of large language models before tracing the academic community's attempts over the past year to fuse visual perception with language capabilities. The core of the talk introduces CogVLM, a 17 billion parameter open-source multimodal model comparable in efficiency to a 7B dense model, and CogAgent, a model built for GUI navigation and OCR tasks. Ding closes by surveying applications of multimodal models and open research directions. The talk assumes familiarity with transformer architectures and is pitched at students and researchers already versed in the CS25 course's core material on attention and language modeling, with Ding drawing on his own work on CogView, CogVideo, and GLM.