Hacker News Reader

Hacker News

Comment by mietek | Hacker News Reader

Parent Full thread

mietek

> This works great for things with patterns, but it doesn't work with language.

Sure it does.

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

> Hawkins' mantra is sequence based memory.

Note that Hawkins wrote “On intelligence” in 2005.

Schmidhuber wrote “Learning complex extended sequences using the principle of history compression” in 1992, and “Long short-term memory”, with Hochreiter, in 1997.

Today, a LSTM-based RNN can answer your Gmail.

http://people.idsia.ch/~juergen/rnn.html

http://googleresearch.blogspot.com/2015/11/computer-respond-...

View on HN

Sorry! To clarify that it doesn't work with language, I mean you can't ask it a question and then expect it to come up with an answer. For most meaningful questions, you can't sequence an answer from it's question in a general way. This might be in part because Hawkins isn't trying to mimic the brain, but learn about how it works to apply the algorithms it uses to our needs/wants.

you can't ask it a question and then expect it to come up with an answer

Why not? If the model was trained on a large enough corpus of texts, it will have seen lots of answers to your question, or similar questions. It can, in principle, extract the important features of those answers, and present them to you as an answer. This is pretty much how the majority of humans would answer "the most meaningful" questions.

Nupic showcases it's "What did the fox eat" use case which uses cortical.io's SDRs of words. In this example it "asks the question" what does the fox eat after teach it what other animals eat, and without ever seeing fox before, it's able to accurately say what it eats since it groups foxes with coyotes, etc, which have a similar SDR. However, this is a sequence... They fed it sentences only in the form "a eats b" and then fed it "fox eats", and it replied "rodent" or whatever... The SDR's have to be sequences in order for the HTM to work. So if you ask "Guess how many balls are in this jug", you have to do an implied computation to guess how many possible balls fit inside the jug, and that implied computation is not the word sequence (or sdr sequence). Even if you gave it an infinite amount of similar questions and answers like this, it would never be able to figure out a general way to answer the question, which means you could always stump it by giving a different sized jug + different sized ball.

H in HTM stands for "Hierarchical". This means, that it can, in principle, construct more general, more abstract SDRs (patterns) based on many low level SDRs it sees. Thus it's plausible to see forming of ideas out of sentences, or something similar. This is how it can build a world model, and then run your input through multiple levels of abstraction, as many as needed to give a high confidence answer.

My example was also spoon feeding questions and answers to the HTM. If you were to just give it a corpus of text, and then ask it a question about that text, it would not be able to formulate an answer that has any meaning, and follow proper English grammar. If you disagree, you can respond with a working example.

If you disagree, you can respond with a working example.

I am the working example. I can read texts and answer questions. HTM is an attempt to explain how I do that. Obviously, it's an incomplete theory, but its main ideas make sense to me. You got a better theory?

I don't have a better theory, but that wasn't my point. It may be that there is a setup to read and understand language, but there has never been code presented based on this technology that does it. This is why I'm interested to see how you could set it up to make it work. If you could prove me wrong, then great! I'd love to see it.

HTM is a theory. It's very incomplete at this point, because our understanding of the brain is very incomplete. Numenta published their code as they discover more and more about the brain. Once the system is complete enough to process language, the code will be able to process language.

Reply on news.ycombinator.com