Comment by zackmorris | Hacker News Reader

Comment by zackmorris | Hacker News Reader

Note that using genetic algorithms to come up with initial weights for neural networks was the state of the art in the late 90s/early 2000s. So this paper is not as novel as it seems, but it's good to have it for reference.

My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training. Hobbyists have access to so much processing power with GPUs now that they can explore techniques that weren't practical for experts 20 years ago. So we may see training speed increase by a few orders of magnitude using techniques besides gradient descent (maybe quantum computing someday, who knows).

The big question though is how to combine networks into hierarchies so that the number of behaviors that can be learned is no longer limited (since pattern recognition is largely a solved problem). I think the way GAs fit in is that they make it much easier to understand and build simple NNs, and possibly train hierarchies or discover topologies that aren't immediately obvious.

Deep learning wasn't novel in 2012 either - it was the removal of a hardware limitation that made it compelling again. I think the same is true for evolving DNNs, but I don't know if the available compute power is there yet.

> My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training.

This is not really true though... even very shallow neural networks can be universal function approximators in a trivial sense because they can be lookup tables, but they are really not expressive enough to generalize well and lack a lot of the expressivity of deep networks.

Shallow networks would probably differ in the number of categories they can differentiate with accuracy.

> shallow neural networks can be universal function approximators in a trivial sense because they can be lookup tables, but they are really not expressive enough to generalize well

You've got it flipped. If anything, shallow neural networks (of an equivalent number of parameters) are more "expressive" than deep networks, BUT that expressivity just makes them overfit. This is the bias vs. variance tradeoff. If anything, deep networks encode our prior belief that there is a hierarchy of features / a compressed representation, which limits the model that is learned, to a model conforming to those priors.

The 'curse of dimensionality' should be required reading https://en.wikipedia.org/wiki/Curse_of_dimensionality for anyone getting into genetic algorithms or deep learning.

Also previously on HN http://www.visiondummy.com/2014/04/curse-dimensionality-affe...

The visiondummy article is great, but how many dimensions are too many for a DNN? Let's say I have 100 TB of image data with 50 classes. Satellite data, I can have huge images, or tile them into small images. He's arguing for smaller down to a point? What point? When the convergence gets to a minimum?

My feeling is that since shallow networks can be made to have equivalent accuracy to deep networks, that the real challenge isn't topology but training.

In a sense, topology is training. One reason that DNNs outperform shallow networks for so many problems is that presumably their topology captures structure inherent in the world that would otherwise have to be discovered by a simpler network.

I'm new to the field. What makes you say that shallow and deep networks have equivalent accuracy? Is there a recent publication/experiment I missed? Thanks!

https://arxiv.org/abs/1703.02660

Disclosure: I contributed to the linked work.

To do simple tasks, you don’t even need neural networks, in short. If you want to do complex tasks, training the neural net isn’t the problem, but a NN can indeed represent more than say a linear policy. The issue is getting the NN to do the complex thing correctly: it’s more of an exploration problem than function approximation problem.

> To do simple tasks, you don’t even need neural networks, in short.

Ssh! We've got a gold rush to maintain.

I had to do this for a homework assignment about 10 years ago!

Reply on news.ycombinator.com