Hacker News Reader

Comment by michael_dorfman | Hacker News Reader

Anyone have a clue as to why (or, indeed, whether) Bayesian mathematics work any better than the more old-school neural network algorithms?

View on HN

The reasons why the neural network model failed and Bayesian inference is much more applicable are described very well by Jeff Hawkins in his book "On Intelligence" (http://onintelligence.com/). I highly recommend it, if just for the the questions and trains of thoughts it raises.

Honestly, I think it just comes down to simplicity. Bayesian algorithms tend to make fewer assumptions a priori. By contrast, look at something like backprop.

I respectfully disagree:

Bayesian methods do not necessarily make fewer assumptions. They just allow you to make your assumptions explicit and to evaluate how sensitive your conclusions were to those assumptions. Conventional statistical methods all have ways of doing this but they are ad hoc. In this sense you are totally correct - bayes is simpler.

Also because bayesian inference in any interesting problem domain is often computationally impossible, you must make approximations - which adds even more complexity.

Backpropagation on the other hand, is remarkably simple -- it's just the chain rule from calculus.

this was interesting.

is bayesian compuatation for "big interesting" problems NP-complete? learning them is? http://www.google.se/search?client=firefox-a&rls=org.moz... http://citeseer.ist.psu.edu/120871.html

then what does the brain do? approximations as well? or bayesian networks are just a good model of the brain but far from complete? does it suggests in some similar way to the http://en.wikipedia.org/wiki/Uncertainty_principle that there is a limit to what we can know in this aspect(if thats what the uncertainty principle suggests for that part of physics?)?

where is this Eliezer Yudkowsky guy? come and give your opinion please!

I think any similarities between the uncertainty principle in physics and limits of what we can know about our minds is poetic or formal.

There might be some rule which the uncertainty principle, the mind problem, and Godel's incompleteness fall out of, or that relates them eloquently and definitively at a higher order of abstraction, but they aren't the same.

I think there is a common kind of error equivocating the hard problem of consciousness (which is a formal way of understanding your "limit to what we can know in this aspect") with uncertainty principle or Godel's Incompleteness (another idea that looks like and is confused with the uncertainty principle or the hard problem of consciousness if you squint right). It is like recognizing y=2X^2 and y=2X^2+2 have the same derivative, but using this insight to argue that 2X^2=2X^2+2.

good points. more on the subject:

http://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_t...

Minds and machines

    Main article: Mechanism (philosophy)#Gödelian arguments

Authors including J. R. Lucas have debated what, if anything, Gödel's incompleteness theorems imply about human intelligence. Much of the debate centers on whether the human mind is equivalent to a Turing machine, or by the Church-Turing thesis, any finite machine at all. If it is, and if the machine is consistent, then Gödel's incompleteness theorems would apply to it.

Hilary Putnam (1960) suggested that while Gödel's theorems cannot be applied to humans, since they make mistakes and are therefore inconsistent, it may be applied to the human faculty of science or mathematics in general. If we are to believe that it is consistent, then either we cannot prove its consistency, or it cannot be represented by a Turing machine.

http://en.wikipedia.org/wiki/Mechanism_%28philosophy%29#G.C3...

If the brain uses bayesian inference, could emotions and intuitions perhaps be functionally equivalent to the approximations needed to do the impossible computations you are talking about?

That's an interesting thought.

people who froze up when confronted with incomplete information about probabilities of outcomes were out competed by those who did something, even if they chose wrong a lot of the time.

"In this sense you are totally correct - bayes is simpler."

That's exactly the sense I referring to and similarly the error correction terms (i/o) in backprop.

But I mis-read the original question: Either "works" just fine - it's just that there seems to be a movement away from NN and toward Bayes. That's what I was answering even as it's tough to judge that right now. Still, NN seemed much hotter 10 years ago in the literature. Where's Google Trends for academic papers?

Reply on news.ycombinator.com