Hacker News Reader

Comment by akuma73 | Hacker News Reader

What about XOR?

From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

View on HN

The simple answer is that we no longer use the standard linear perceptrons that McCulloch and Pitts put forth in 1943. The limitations of perceptrons and their training algorithms are well-known and well-studied, and in the last 60 years we've come a long way.

A book came out a little bit ago which covers basic neural networks, and culminates in showing a network that can estimate XOR [1]. I strongly recommend reading not only the chapter, but the whole book.

[1] http://natureofcode.com/book/chapter-10-neural-networks/

'Single-layer' is an easily overlooked and absolutely essential modifier. Very few networks these days are single layers. (Three is the minimum for infinite dimensional functional approximation, given nonlinear activations.) Deep Networks you see in new research papers these days have, at minimum, perhaps three layers. Some by LeCun et al. will go as high as nine or twelve layers, with four or sixteen layers in breadth.

I'm pretty sure linearity was also part of the picture -- no matter how many layers of neurons you have, as long as they perform linear combinations, they could all be replaced with a single one (ignoring numeric errors).

I mentioned this in my post ("given nonlinear activations"), but you're correct to point out its importance. The combination of linear functions is always linear. Full stop. You need at least one non-linear layer to get reasonable results.

If you join a bunch of perceptrons together that limitation goes away. Another path is to make the problem effectively linear again by transforming into higher dimensions, kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly.

> kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly

Minsky and Papert hate him! Use this one weird trick ….

(Sorry; I couldn't resist!)

Sorry, you were beaten to the joke a long time ago: http://www.oneweirdkerneltrick.com/

Neural nets now use non-linear functions so they can do things like xor.

"Single-layer" is an important restriction there, however.

http://en.wikipedia.org/wiki/Feedforward_neural_network#medi...

Reply on news.ycombinator.com