It's fascinating that none of the comments mention neural network applications.
Seems like these people came a bit too early to the party.
Seems like these people came a bit too early to the party.
Some people have left biomimicry in the dust and are manipulating matrices to sets find optimal parameter sets representing concepts.
Other people (https://www.humanbrainproject.eu/en/) haven't! There is a big community of people building novel types of neural network using spikes and so on.
And you are right - it isn't clear who is right. On one side the "no nature" folks might say : we don't even understand neurons to the point where we can definitely categorize them in a mouse brain, let alone explain their behaviour, or actually properly measure their behaviour, and so on and so on - this before understanding any of it really.
On the other hand the "lets mimic things we don't understand" folks would say - we need to understand this because the human brain out performs all AI today on a 20 watt budget.
My view is the understand stuff agenda is right - but not now and probably not for 20 years (in a big way). We need small scale capability and direction finding work - and we will need to do that for a long time.
The ReLU was originally introduced by computational neuroscientists as a more biologically accurate approximation compared to the other activation functions. E.g. https://www.nature.com/articles/35016072
I would expect similar history behind convolutional neural networks, given how the early stages of the vision system have been understood to operate.
I wouldn’t dismiss an approach because it isn’t popular right now. Biological neurons force you to think about constraints you normally wouldn’t and crucially have solved temporal credit assignment, memory etc. in interesting ways that already have inspired some breakthroughs.
As an example the replay memory idea was directly inspired by the rough understanding neuroscientist developed of the role of the hypocampus in learning.
(The activation function is the neuron('s mathematical model) - we elide everything that happens internal to the neuron and model it as only its inputs and outputs.)
The main problem is that the neurons in this article are not trainable (you can't update the weights dynamically) and not differentiable (you can't use the usual gradient-based training procedures to update the weights). They could be used for inference only though.