Am I crazy, or is the short-run benefit more for unsupervised feature learning more than classification accuracy? If I recall from my brief reading on the subject, kernel-aided SVMs and Random Forests are basically equivalent to a three-layer deep graphical net, but on painfully feature-engineered inputs. I'd wager learning features is more useful now than any benefit of a super-deep architecture.

(I would love for a practitioner to shoot this down or corroborate my hunch-- awfully new to neural nets, having basically read a monograph, played with Theano, and watched some Hinton lectures.)

SVMs may still be useful as a classifier sitting on top of a deep model. Plugging the features learned during unsupervised pre-training into an SVM instead of a neural net is perfectly valid.

Lower layers of the "deep" model are (typically) performing non-linear dimensionality reduction. i.e. generating a set of high-level features which make subsequent classification easier than it would have been on the raw input data.

You can use SVMs for deep learning, for one.

http://books.nips.cc/papers/files/nips25/NIPS2012_1290.pdf