We explore this interpretation for regularization: there is inherent statistical inefficiency in empirical risk minimization and good regularizers reduce this inefficiency. This viewpoint is in contrast to regularization to reduce model complexity, or to introduce a priori knowledge into the model. We show sufficient conditions under which there is a Stein phenomenon in empirical risk minimization, and we correct for this phenomenon using a shrinkage operator, which we interpret as a regularizer.

Joint work with Alan Mackey.