Very interesting and well written paper. They show a deep-neural network (which people thought had more explaining-power-per-parameter) can be "emulated" by a shallow neural network with the same number of parameters.

What Ba and Caruana have shown is that deep neural networks---as architecture---are not more powerful. The previous general opinion (at least this is what I thought) was that deep architectures of the form:

  predictions
  layer 3  features of features of features
  layer 2  features of features 
  layer 1  features 
  data layer  

would be more efficient (i.e. more explaining power per parameter). Nope. Apparently, a shallow neural network of the form

  predictions
  layer 1  features   (a lot more of them)
  data layer  

with an equal number of units in layer 1 as the total number of units in deep network, can obtain the same accuracy. Thus, deep neural networks and shallow neural networks have roughly equivalent explaining-power-per-parameter.

A separate result shows deep convolutional neural networks can be simulated by shallow NNs with a simulation overhead of ~10x.

Another cool observation they make about "installing" the parameters of ML models. We'll use the following analogy to illustrate the point:

  ML model               <==>  hardware 
  ML model parameters    <==>  software 

Running program X involves (1) buying/renting a server, (2) compiling X for your architecture, and (3) running X. Of course there is always the alternate step (2') of installing pre-compiled binaries.

Similarly, we can think of using a ML model as a three step procedure (1) ML.init() (malloc for Model.parameters), (2) Model.train() (fit parameters to training data), (3) Model.predict(new_datum).

The authors point out the existence of a possibility of distributing models parameters as binaries. They demonstrate an alternate step (2') in which they train a ''FancyModel'', then set the parameters of the simple model so it "simulates" the FancyModel: ''Model.parameters = simulate(FancyModel)''. In other words, the model parameters of a ML model need not be learned through the model itself.

What I find fascinating is it seems deep architectures don't have any more representational power, but they do have more learning power. Something very interesting is going on...