Hacker News Reader

Comment by lutorm | Hacker News Reader

So about these deep neural networks: I seem to remember from some introductory AI class many years ago that a multilayer neural network can't do anything a single-layer one can't. So what's the deal here then? Is it because the algorithms for learning can't be reduced to an equivalent single-layer network?

View on HN

The argument is that single-layer networks would need to be impractically wide to have the same representational power as a narrower network with multiple layers.

I haven't looked into the math myself, but apparently there are situations where the increase in representational power for networks of a given size can be increased exponentially by moving nodes into higher layers.

In theory, at least, a deep architecture can learn the same things as a shallow one, but with less nodes. See http://www.iro.umontreal.ca/~lisa/publications2/index.php/pu...

There isn't anything a large number of if-then statements can't do either. Why do programmers use nested if-then statements or states?

For some GPUS, all code runs, no nested its, just flags to turn unused paths into no-ops. All I'm trying to say is even in your example, the optimal thing we could come up with depended on the problem at hand (graphics vs general purpose). Newer GPUs are different, but still, the ridicule of his question isn't appropriate, reasoning by analogy is error prone, and the real answer to his question is probably a lot more rigorous.

This is because GPUs are parallel and can't do nested ifs, not because representing code that way is efficient.

Even GPUs do sequential operations. You could never program the vast majority of algorithms like "if the input is exactly 100011110101... then output 100011001...".

If that seems silly, that is exactly how the proof that single layer NNs are perfectly general works. It proves that you can represent any series of if then statements like that in an NN. And people who don't understand the proof are mislead into thinking single layer NNs are just as good as deep NNs.

Reply on news.ycombinator.com