pakl

I take the point to be that there aren't "deeper" fundamental principles at play in these models. Tremendous progress has comes from simply tweaking of the numbers of layers, or how the feed forward to each other (skipping layers, etc), or by throwing more computer power or data at the same basic algorithm.

Where might we look for deeper principles? One idea is to consider what brains do and how they might be doing it. (I'm not saying we need to go down the rabbit hole of biological detail -- on the contrary I'm suggesting we look at known or even hypothesized principles of brain operation and import them into AI.)

Two ideas we have used in our work: prediction (over time), recurrent feedback (most brain regions have more feedback than feedforward inputs)

What you mention in point #1 isn't the main issue with games, but with the "reactive" approach in AI. Reactive AI systems are necessarily going to be more limited than say, predictive AI systems (a new emerging class of AI models). [1]

Agreed, though, that the focus on games is a long-standing source of problems holding back AI. My colleague has referred to this as "ludic AI" in a recent blog post where he expands on your statement about the real challenge of intelligence. [2]

[1] Reactive vs Predictive AI (http://blog.piekniewski.info/2016/11/03/reactive-vs-predicti...)

[2] Learning Physics (http://blog.piekniewski.info/2016/11/30/learning-physics-is-...)

Well, it seems taking a principled approach helps avoid the WTFs :)

Please see my other comment for informative video links reviewing recent work by community collaborators.

For some recent information (mid/late 2016) on applying linguistic methodology to the Voynich Manuscript, see this two part video.

It's really great to see how much progress the community has made.

https://youtube.com/watch?v=4cRlqE3D3RQ

https://youtube.com/watch?v=8nHbImkFKE4

Just define consciousness, and then maybe we can start to have a scientific discussion about it. ;)

To see some concrete examples of how this sort of phenomenon causes surprising failures in deep networks classifying images, see "Intriguing properties of neural networks"[1]

[1] https://cs.nyu.edu/~zaremba/docs/understanding.pdf

Yes, but in direct contrast to Chomsky (who would say there's not enough data/time for kids to learn from) I am saying that there is a ton of rich dynamical data in the world around us all the time. Plenty to learn from.

Just plug a webcam into an adequate system and allow it to learn dynamics by trying to predict what it will see next.

Chomsky is almost right for the wrong problem: there won't ever be enough human-labeled data for good generalization. ;)

Exactly! Now consider that deep networks that classify images are tasked with getting reliable statistics in very high dimensional spaces. Conv nets are being forced to map from high dimensional spaces down to a very low dimensional categorical decision. This is why weird classification errors you see in "adversarial examples" keep popping up. The learned classification boundaries are very spiky, and it's easy for the world to fall in between the spikes. More data can't solve it (at least not practically) because there are way too many gaps between the spikes.

A more tractable approach is to learn to use dynamics for perception rather than ("just") statistics. The dynamical physics of a ball rolling is much simpler (lower dimensional, more tractable) than a statistical view of millions of differently illuminated pixels hitting a camera.

(A colleague of mine has a blog post on this issue of "statistics and dynamics" at http://blog.piekniewski.info/2016/11/01/statistics-and-dynam...)

I commend you. Very few people actually try feeding images into these things to show how bad they are.

For a real shock at how poor performance is, try feeding frames of video in. (Video is different because it generally doesn't have carefully stereotypically framed, exposed, in-focus content).

Edit: examples of ResNet applied to video by a colleague http://blog.piekniewski.info/2016/08/12/how-close-are-we-to-...

> There are too many hard cases for vision.

Even worse, most machine learning vision approaches make the vision problem much harder on themselves. They do not treat the visual world as the dynamic physical interacting processes that give rise to it.

A disadvantage treating of vision as static frames of pixels is that deep feedforward networks have to "memorize" all the physical dynamical effects of e.g., shadows on textures.

Such systems cannot generalize well. A more promising approach is hierarchical systems with ubiquitous recurrent connectivity.

Agreed -- If you get away from traditional feedforward networks by adding recurrence throughout, then at least there is some chance of learning scale-free features and compositionality.

You've hit on a key insight. Predicting the future (even by a bit) turns out to be a very powerful learning signal for building models of the world.

It won't work on a traditional feedforward neural network but if you have feedback everywhere it appears to work.

If one wants general AI that can deal with/understand the world, the system needs to learn based on (raw, unadulterated) data from the world. These data are highly dynamic and rarely fit neatly into human-labeled categories. This is part of why the currently-hyped supervised pattern recognition is not all that helpful for general AI.

By the way, the above requirement of real world data probably even applies to building chatbots. (A successful chatbot will need some understanding of the world it is talking about; this is what researchers mean when they say "grounding" is important for NLP).

There are some promising ideas out there based on unsupervised ("self-supervised") learning. There, the problem of needing big labeled datasets doesn't exist: just turn on a camera and have a motor point it at the surrounding world. Data of this kind is and always has been super abundant. But using it requires making a conceptual leap away from supervised pattern recognition -- which is pretty much what today's "AI" does...

I'm optimistic about what'll happen right after the upcoming AI winter. :)

So long as AI remains tasked with categorizing human-taken photos or playing human-created games (no matter how "complex"), AI will remain just that -- artificial -- and not "real".

Given the rate of hype that you point out, I suspect a winter of some kind will hit before enough folks realize this.

[edited: reorder]

Interesting point about reverse-engineering the brain possibly succeeding first. The result of logical AI, however, is likely to be fundamentally different from the human brain or human mind. (Recall how difficult even simple logic is for... pretty much everyone.)