Foundations of Deep Learning

May 23–Aug. 9, 2019

Deep learning is the engine powering many of the recent successes of artificial intelligence. These advances stem from a research effort spanning academia and industry; this effort is not limited only to computer science, statistics, and optimization, but also involves neuroscience, physics, and essentially all of the sciences. Despite this intense research activity, a satisfactory understanding of deep learning methodology — and, more importantly, its failure modes — continues to elude us. As deep learning enters sensitive domains, such as autonomous driving and health care, the need for building this kind of understanding becomes even more pressing.

The goal of this program was to address this need by aligning and focusing theoretical and applied researchers on the common purpose of building empirically relevant theoretical foundations of deep learning. Specifically, the intention was to identify and make progress on challenges that, on one hand, are key to guiding the real-world use of deep learning and, on the other hand, can be approached using theoretical methodology.

The program focused on the following four themes:

Optimization: How and why can deep models be fit to observed (training) data?
Generalization: Why do these trained models work well on similar but unobserved (test) data?
Robustness: How can we analyze and improve the performance of these models when applied outside their intended conditions?
Generative methods: How can deep learning be used to model probability distributions?

An integral feature of the program was bridging activities that aimed to strengthen the connections between academia and industry. In particular, in addition to workshops and other weekly events, the program hosted weekly bridging days that brought together local Bay Area industry researchers and regular program participants.

This program was supported in part by the Patrick J. McGovern Foundation.

List of Weekly Visitors:
Anima Anandkumar (California Institute of Technology and Nvidia), Yasaman Bahri (Google Brain), Samy Bengio (Google), Paul Christiano (OpenAI), Shalini De Mello (NVIDIA), Inderjit Dhillon (Amazon), Vitaly Feldman (Google Brain), Jonathan Frankle (Facebook), Mohammad Ghavamzadeh (Facebook AI Research), Dan Hill (Amazon), T.S. Jayram (IBM Almaden Research), Tomer Koren (Google Research), Ming-Yu Liu (Nvidia), Philip Long (Google Brain), Nimrod Meggido (IBM Almaden Research Center), Ofer Meshi (Google), Ilya Mironov (Google Brain), Hossein Mobahi (Google), Qie Hu (Amazon), Jakub Pachocki (OpenAI), Rina Panigrahy (Google Brain), Maithra Raghu (Google Brain), Nima Reyhani (AirBnB), Sujay Sanghavi (Amazon), Sam Schoenholz (Google Brain), Hanie Sedghi (Google Brain), Rajat Sen (Amazon), Szymon Sidor (OpenAI), Yoram Singer (Google Brain), Jascha Sohl-Dickstein (Google Brain), Kunal Talwar (Google), Felix Juefei Xu (Alibaba Group), Laura Zaremba (Groq), Kai Zhong (Amazon)

Organizers:

Samy Bengio (Google), Aleksander Mądry (Massachusetts Institute of Technology), Elchanan Mossel (Massachusetts Institute of Technology), Matus Telgarsky (University of Illinois, Urbana-Champaign)

Long-Term Participants (including Organizers):

Raman Arora (Johns Hopkins University), Peter Bartlett (UC Berkeley), Misha Belkin (Ohio State University), Shai Ben-David (University of Waterloo), Samy Bengio (Google), Constantinos Daskalakis (MIT), Ilias Diakonikolas (University of Southern California), Alex Dimakis (University of Texas at Austin), Alyosha Efros (UC Berkeley), Laurent El Ghaoui (UC Berkeley), Noureddine El Karoui (UC Berkeley), Dylan Foster (Massachusetts Institute of Technology (MIT)), Suriya Gunasekar (Toyota Technology Institute, Chicago), Boris Hanin (Texas A&M), Moritz Hardt (UC Berkeley), Daniel Hsu (Columbia University), Varun Jog (University of Wisconsin, Madison), Michael Jordan (UC Berkeley), Adam Klivans (University of Texas at Austin), Jason Lee (University of Southern California), Po-Ling Loh (University of Wisconsin, Madison), Tengyu Ma (Stanford University), Aleksander Mądry (Massachusetts Institute of Technology), Michael Mahoney (International Computer Science Institute and UC Berkeley), Elchanan Mossel (Massachusetts Institute of Technology), Christos Papadimitriou (Columbia University), Ali Rahimi (Google), Benjamin Recht (UC Berkeley), Dan Roy (University of Toronto), Sushant Sachdeva (University of Toronto), Anant Sahai (UC Berkeley), Colin Sandon (Massachusetts Institute of Technology), Peter Sarnak (IAS), Johannes Schmidt-Hieber (University of Twente), Dana Scott (UC Berkeley), Aarti Singh (Carnegie Mellon University), Mahdi Soltanolkotabi (University of Southern California), Dawn Song (UC Berkeley), Daniel Soudry (Technion - Israel Institute of Technology), Nati Srebro (Toyota Technological Institute at Chicago), Nike Sun (Massachusetts Institute of Technology), Matus Telgarsky (University of Illinois, Urbana-Champaign), Rene Vidal (Johns Hopkins University), Nisheeth Vishnoi (Yale University), Martin Wainwright (UC Berkeley), Rachel Ward (University of Texas at Austin), Amir Yehudayoff (Technion Israel Institute of Technology)

Research Fellows:

Yossi Arjevani (Weizmann Institute of Science), Yu Bai (Stanford University), Gintare Karolina Dziugaite (University of Cambridge), Soheil Feizi (University of Maryland, College Park), Surbhi Goel (University of Texas at Austin), Quanquan Gu (University of California, Los Angeles), Pritish Kamath (Massachusetts Institute of Technology), Qi Lei (University of Texas at Austin), Behnam Neyshabur (New York University), Quynh Nguyen (Saarland University)

Visiting Graduate Students and Postdocs:

Bolton Bailey (University of Illinois Urbana-Champaign), Tianle Cai (MIT), Xiang Cheng (UC Berkeley), Logan Engstrom (MIT), Ruiqi Gao (University of Southern California), Suprovat Ghoshal (Indian Institute of Science), Andrew Ilyas (MIT), Ruhui Jin (University of Texas at Austin), Matt Jordan (University of Texas at Austin), Stefani Karp (Carnegie Mellon University), Saleet Klein (Massachusetts Institute of Technology (MIT)), Frederic Koehler (MIT), Gaurav Mahajan (UC San Diego), Omar Montasser (TTI-Chicago), Adit Radhakrishnan (MIT), Sujit Rao (Massachusetts Institute of Technology), Shibani Santurkar (Massachusetts Institute of Technology), Adam Sealfon (Massachusetts Institute of Technology), Jonathan Shafer (), Dimitris Tsipras (Massachusetts Institute of Technology), Kiran Vodrahalli (Columbia University), Colin Wei (Stanford University), Shirley Wu (University of Texas at Austin), Ruicheng Xian (University of Illinois Urbana-Champaign)

Workshops

Deep Learning Boot Camp

Tuesday, May 28–Friday, May 31, 2019

Organizers:

Sébastien Bubeck (Microsoft Research; chair), Elchanan Mossel (Massachusetts Institute of Technology), Matus Telgarsky (University of Illinois, Urbana-Champaign)

Frontiers of Deep Learning

Monday, Jul. 15–Thursday, Jul. 18, 2019

Organizers:

Aleksander Mądry (Massachusetts Institute of Technology; chair), Samy Bengio (Google), Tengyu Ma (Stanford University)

Emerging Challenges in Deep Learning

Monday, Aug. 5–Thursday, Aug. 8, 2019

Organizers:

Matus Telgarsky (University of Illinois, Urbana-Champaign; chair), Moritz Hardt (UC Berkeley), Sergey Levine (UC Berkeley), Aleksander Mądry (Massachusetts Institute of Technology), Ohad Shamir (Weizmann Institute)

Deep Learning Reunion

Monday, Aug. 10–Thursday, Aug. 13, 2020

Organizers:

Samy Bengio (Google), Aleksander Mądry (Massachusetts Institute of Technology), Elchanan Mossel (Massachusetts Institute of Technology), Matus Telgarsky (University of Illinois, Urbana-Champaign)