As someone with only basic idea of deep learning, I want to ask this question:
If deep learning is mimicking how our brains work, wouldn't it be theoretically as powerful as our brains? In terms of architecture and complexity of the model.
If deep learning is mimicking how our brains work, wouldn't it be theoretically as powerful as our brains? In terms of architecture and complexity of the model.
We aren’t mimicking how our brains work in any broad sense. At best it’s a very narrow definition.
In this case - the larger the matrices - the more you can do. There’ll probably be a Moore’s law of AI at some point.
Why would you assume that? I did a cursory GIS and Bing search, and somewhere between 1-5% of the returned images were bunless wieners.
Literally the first image returned for hot dogs is a bunch of bunless ones on a grill. There are plenty more.
For comparison, I tried out Clarifai's image classification API: https://clarifai.com/demo It has no problem classifying some bunless ones on a grill (https://www.hot-dog.org/sites/default/files/2016-09/sausage%...) as being sausage/pork/beef/hotdog, with nary a 'sandwich' to be seen, while conversely, a Big Mac (https://d1nqx6es26drid.cloudfront.net/app/uploads/2015/04/04...) gets sesame/burger/lettuce/sandwich/bun/bread tags with no sausage/bratwurst/hotdog to be seen. Clarifai's NNs seem to do fine on 'hot dog or not hot dog'...
the idea originally was that a neuron can be roughly modeled as something that sums up inputs from other neurons and then outputs some kind of threshold/activation of those inputs. this is probably sort of true of biological neurons although as living cells they are much more complicated than that.
but lots of recent and highly successful techniques in deep learning have given up on even that very weak amount of biological imitation. One simple example: lots of techniques involve multiplying together the outputs of "neurons". There is no sensible biological interpretation of this operation.
Neural networks are Turing-complete (assuming all the usual pedantic assumptions you need to make that statement meaningful) so in that sense they are as powerful as our brains.
The other way is more controversial, but I don't think it's outrageous to speculate that a brain could be accurately simulated by a Turing machine.
Fair enough. I got a different impression from reading your original comment :)
That's of course true, but then most computers aren't that similar to a classic Turing machine either. The idea of Turing machines is that they can simulate anything else that we mean when we say "computation".
> Can a Turing machine simulate a brain - as in, create intelligence like the one our brains expose?
Yes, it can. I mean, if it can't, that means there's some notion of "computability" that a Turing machine doesn't capture - this would violate the "Church-Turing" thesis. Such a thing is possible, of course, since we have no proof that a Turing machine is the limit of what we mean by "computable" - but it would be a completely revolutionary fact in Computer Science, and one that is very unlikely, considering how many years people have been trying to come up with alternative models of "computation" that all turn out to be equivalent to Turing machines.
-
The human brain isn't magic - it works according to the laws of physics like anything else. Either it is simulatable by a Turing machine, which would make the most sense, or it isn't, in which case we're missing something about how you can compute things, which would be surprising and amazing.
Take for example machine vision. ConvNET is inspired by the lowest levels of human visual cortex and there are analogies to be drawn, but no direct mimicry.
https://neurdiness.wordpress.com/2018/05/17/deep-convolution...
Neurons are families of cells that come in many shapes and most of the time get oversimplified for didactic purposes. The processes involved in neuroscience are also oversimplified most of the time.
It is often said that dendrites are the input, axons the output, and that spikes are fired when a threshold is met... but that's not always true. The biological situation is full of special cases.
Then you have the way the network is formed. The amount of neurons and layers in a deep neural networks are just hardcoded in advance. That's not the case in biology.
Then you have volume: biological neural networks have many more neurons, and many more connections between those. For example each Purkinje cell (in the cerebellum) can have over 200k dentritic connections.