I will consider the goal of designing deep learning systems that have strong, verified assurances of correctness with respect to mathematically-specified requirements. I will describe some challenges for achieving verified deep learning, and propose a few principles for addressing these challenges, with a special focus on techniques based on formal methods. I will illustrate the ideas with examples and sample results from the domain of intelligent cyber-physical systems, with a particular focus on the use of deep learning in autonomous vehicles.