This is an important update for practitioners: beware of Adam, it has fundamental flaws. In practice, carefully tuned learning rate schedules for basic SGD, as discussed in this, has gotten me much better results on very large very complicated data sets... SGD is clearly a brute force approach requiring more effort by the programmer, but with sufficient data and sufficient computing power it's still the best starting point, because it assumes the least about the constraints of the data.

As a related aside, my current interest is on stable efficient use of higher order derivatives for topological inference in optimization, and I wish more energy was invested in such math...