Keynotes

The recent success of large language models has been characterized by scaling laws " the power law relationship between performance and training dataset size, model parameter size, and training compute. In this talk, we will discuss ways to push the scaling laws even further by innovating across data, models, software and hardware. This includes reinforcement learning from human and AI feedback to improve learning efficiency, sparse and dynamic mixture-of-experts neural architectures for better performance, an automated framework for co-designing custom AI accelerators, and a deep RL method for chip floorplanning used in multiple generations of Google AI's accelerator chips (TPU). Through these cutting-edge examples, we will outline a full-stack approach that leverages AI to overcome the next set of scaling challenges.