Training a deep belief network on a GPU with 100 million free parameters
ai.stanford.edu
ai.stanford.edu
"Deep learning" is the new big trend in Machine Learning. It promises general, powerful, and fast machine learning, moving us one step closer to AI.
An algorithm is deep if the input is passed through several non-linearities before being output. Most modern learning algorithms (including decision trees and SVMs and naive bayes) are "shallow".
For intuition, imagine if I told you that your main routine can call subroutines, and your subroutines could call subsubroutines, but you couldn't have any more abstraction than that. You can't have subsubsubroutines in your "shallow" program. You could compute whatever you wanted in a "shallow" program, but your code would involve a lot of duplicated code and would not be as compact as it should be. Similarly, a shallow machine learning architecture would involve a lot of duplication of effort to express things that a deep architecture could more compactly. The point being, a deep architecture can more gracefully reuse previous computations.
Deep learning is motivated by intuition, theoretical arguments from circuit theory, empirical results, and current knowledge of neuroscience. Here is a video where I give a high-level talk on deep learning, and describe this intuition: http://bilconference.com/videos/deep-learning-artificial-int...
Here is more detailed information about deep learning: http://deeplearning.net/tutorial/
Deep learning techniques can typically be expressed as dense matrix operations, making them suitable to being sped up using GPUs. Which makes deep learning even more powerful.
They are also more capable of dealing with tons of variables (in this case, ~100 million). If you had 100 million yes/no values that needed to be checked, could you really answer all of them accurately? In some cases, sure, but not many, and what's the correct number of variables for simulating a language, and what do they relate to? An unsupervised network will try to fit them automagically, rather than taking your word for it.
Unsupervised learning also poses a lot of intelligence-modeling uses, because we're essentially unsupervised for some of the "hardest" tasks to learn. Especially prior to learning to speak. All there really is for feedback is pain, and that doesn't necessarily lead directly to language.
But bravura should be able to answer better / more accurately; hopefully they'll chime in too. I'd like to know if there's more to it than what I know, which is pretty minimal.
The following explanation is an oversimplification to give you intuition into what is going on: The difficulty in training a deep architecture is that, if you use standard backprop, the gradient signal doesn't flow back to the lowest layers. Instead, the few supervised output units pass back a small gradient signal, and at each layer the gradient signal has increasing noise as it gets passed backwards. So the top layers overfit, and the lower layers don't get tuned effectively. The lower layers essentially fire random noise, and the top layers overfit.
For this reason, before 2006, no one knew how to train a deep architecture (besides Yann LeCun, with his convolutional architectures, but those were not as general purpose).
The breakthrough in 2006 came when Hinton came up with the original DBN algorithm. Bengio et al (http://www.iro.umontreal.ca/~lisa/publications/index.php?pag...) followed up by teasing apart the important steps in training a deep architecture. They were: greedy layerwise training, i.e. construct one layer, then the next layer, then the next layer, etc. unsupervised pretraining When you train a single layer using an unsupervised criterion, the gradient signal is passed backwards through only one hidden layer (the layer you are constructing). And the output layer for an unsupervised criterion has as many units as the input. So the output layer passes back a strong gradient signal, and it doesn't have to travel very far.
By doing this unsupervised pretraining in a layer-by-layer manner, the deep network receives a good initialization of its parameters. Then, when finetuning against the supervised criterion using backprop, you can find a better local minimum.
The work of Erhan et al show that the effect of unsupervised pretraining is not only regularization, but also improved optimization. Take a look at there work for a great empirical study with large scale experiments and pretty graphs: http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erh...
p.s. feel free to send me an email to followup.
It's not particularly provable, but it works damn well. And yeah, everyone wants something they can prove... but I side with the "make something useful" crowd before "make something provable".
And fantastic links, thanks! I've been meaning to get further into the field, but I've been swamped, don't know where to start, and have taken the only class on it at my college.
In the 1980s at SAIC, we spent a small fortune designing custom (Harvard memory architecture) hardware to do back-prop neural networks relatively fast, but fast back then was about 23m/sec FP operations. My droid phone could probably do more than that now :-)