Recent advances linking optimization, machine learning, and systems research have enabled the rapid solution of data analysis problems on terabytes of data.  In this talk, I will survey both the algorithmic and computational state-of-the-art, describing some of the bottlenecks and complexities not captured by simple flop or message counting.  I will also highlight new architectures and computing substrates that could potentially enable scalable implementations of algorithms with superlinear scaling, widening the scope of optimization problems we can solve on large data volumes.