
Terminal-Bench: Evaluating AI Agents
Alex Shaw and Ryan Marten of the Laude Institute explain how they built Terminal-Bench, a benchmark of hard command-line tasks now cited in frontier model cards, to measure whether AI agents can do real work. Guest-lecturing in Stanford's CS329Z: Engineering AI Agents, taught by Diyi Yang, Michael Ryan, and John Yang, the two co-creators walk through why evaluations have become the source of truth for agent software, how a terminal environment forces agents to handle realistic multi-step tasks, and the mechanics of writing tasks and designing verifiers that catch reward hacking. They also introduce Harbor, their open-source framework for building agent evaluations and reinforcement learning environments, and discuss the practical work of maintaining a benchmark over time as models improve. The talk gives a concrete, practitioner's view of how frontier labs actually test agent capability rather than a theoretical overview.