LECTURES A GRATIS GLOBAL SERVICE
⌕ SEARCH GRATIS GLOBAL ↗
LECTURES
Terminal-Bench: Evaluating AI Agents
SOURCE: YOUTUBE · NO TRACKING UNTIL YOU PRESS PLAY · TROUBLE PLAYING? WATCH AT THE SOURCE ↗

Terminal-Bench: Evaluating AI Agents

73 MIN · EN · STATUS: [ STREAMING ]
RATE THIS
STANFORD

Alex Shaw and Ryan Marten of the Laude Institute explain how they built Terminal-Bench, a benchmark of hard command-line tasks now cited in frontier model cards, to measure whether AI agents can do real work. Guest-lecturing in Stanford's CS329Z: Engineering AI Agents, taught by Diyi Yang, Michael Ryan, and John Yang, the two co-creators walk through why evaluations have become the source of truth for agent software, how a terminal environment forces agents to handle realistic multi-step tasks, and the mechanics of writing tasks and designing verifiers that catch reward hacking. They also introduce Harbor, their open-source framework for building agent evaluations and reinforcement learning environments, and discuss the practical work of maintaining a benchmark over time as models improve. The talk gives a concrete, practitioner's view of how frontier labs actually test agent capability rather than a theoretical overview.

At a glance

Lecture facts

Runtime compared with the other 387 Computer Science lectures
Runtime1 h 13 m
Compared with Computer ScienceLonger than 57%
Source channelStanford Online (YouTube)