With the advent of higher throughput and more accurate technologies to measure protein properties of interest, such as target binding to a drug, the time for machine learning to act synergistically with protein design is here. The obvious first place to do so is to replace the lab measurements with, for example, a deep neural network based predictive model. Then, one can ask how to invert that model to find desired protein sequences. Naively, inverting this model could be viewed as combinatorial optimization. However, one must take into account possibly heteroscedasctic uncertainty of the predictive model. Calibrating these uncertainties, even in region of the training data, has been tackled, but could be improved. Moreover, "further away" from the training data, the uncertainties are arbitrarily bad. How can we tackle the general design problem when the functions we are optimizing cannot even be trusted?

For open-ended tasks it is often difficult to measure or even define performance. For example, it's unclear what objective I should optimize in order to better understand how I should spend my time or which laws we should pass. I'll describe this problem in the language of learning theory, lay out the approaches that seem most promising to me, and overview some current work.

Policy gradient methods are among the most effective methods in challenging reinforcement learning problems with large state and/or action spaces. However, little is known about even their most basic theoretical convergence properties, including: 1) if and how fast they converge to a globally optimal solution (say with a sufficiently rich policy class); 2) how they cope with approximation error due to using a restricted class of parametric policies; or 3) their finite sample behavior. In this talk, we will study all these issues, and provide a broad understanding of when first-order approaches to direct policy optimization in RL succeed. We will also identify the relevant notions of policy class expressivity underlying these guarantees in the approximate setting. Throughout, we will also highlight the interplay of exploration with policy optimization, both in our upper bounds and illustrative lower bounds. This talk is based on joint work with Sham Kakade, Jason Lee and Gaurav Mahajan.

I'll present a brief overview of some recent work on reinforcement learning motivated by practical issues that arise in the application of RL to online, user-facing applications like recommender systems. These include (a) stochastic action sets; (b) long-term cumulative effects; and (c) combinatorial action spaces. With respect to (c) I will discuss SlateQ, a novel decomposition technique that allows value-based RL (e.g., Q-learning) in slate-based recommender to scale to commercial production systems, and briefly describe both small-scale simulation and a large-scale experiment with YouTube. With respect to (b), I will briefly discuss a Advantage Amplification, a temporal aggregation technique that allows for more effective RL in partially observable domains with low SNR, as often arise in recommender systems.

The typical view of deep learning architectures is that they consist of stacked linear operators followed by (typically elementwise) non-linear operators, with minor additions such as residual connections, attention units, etc. However, a great deal of recent work has looked at integrating substantially more structured layers into deep architectures, such as explicit optimization solvers, ODE solvers, physical simulations, and many other examples. In these examples, any differentiable program can serve as a layer in a deep network, and by properly structuring these problems we can encode a great deal of prior knowledge into the system. In this talk, I will highlight the basic approach behind these structured layers, and highlight some recent advances in the area related to incorporating discrete optimization solvers as layers in deep networks. However, despite their potential advantages, these structured layers raise a number of challenges, especially regarding gradient-based training of the systems. I will discuss these challenges and potential ways forward.

At every level of government, officials contract for technical systems that employ machine learning?systems that perform tasks without using explicit instructions, relying on patterns and inference instead. These systems frequently displace discretion previously exercised by policymakers or individual front-end government employees with an opaque logic that bears no resemblance to the reasoning processes of agency personnel. However, because agencies acquire these systems through government procurement processes, they, and the public, have little input into?or even knowledge about?their design, or how well that design aligns with public goals and values. In this talk I explain the ways that the decisions about goals, values, risk, certainty, and the elimination of case-by-case discretion inherent in machine-learning system design make policy; how the use of procurement to manage their adoption undermines appropriate attention to these embedded policies; and, draw on administrative law to argue that when system design embeds policies the government must use processes that address technocratic concerns about the informed application of expertise, and democratic concerns about political accountability. Specifically, I describe specific ways that the policy choices embedded in machine learning system design today fail the prohibition against arbitrary and capricious agency action absent a reasoned decision making process that both enlists the expertise necessary for reasoned deliberation about such choices and makes visible the political choices being made. I conclude by sketching out options for bringing necessary technical expertise and political visibility into government processes for adopting machine learning systems through a mix of institutional and engineering design solutions.

A key question is how to better leverage the data generated (from RL agents or humans) to get better control policies. We would like to be able to combine the power of extremely expressive function approximators like deep learning with rigorous statistical guarantees to ensure data efficiency and strong guarantees on resulting performance. In this talk I will outline some of our recent work in this space and potential future directions.