Hacker News Reader

Comment by make3 | Hacker News Reader

this would be the dream, but biological neural networks don't do gradient decent, so have little to do with ANNs until we discover how they really train.

View on HN

It would be remarkable if biological neural networks don't use gradient descent. They very likely do even if it is a highly approximate gradient. Neurons are optimizing for some objective, and the gradient is usually a very efficient way to do that optimization. It would be remarkable if evolution missed out on such a thing. Even simple Hebbian learning can be viewed as a gradient descent technique.

Gradients can be calculated in remarkably simple ways. Look at Hinton's recirculation networks for example.

I think you meant to say backprop.

How would brain do gradient descent without backpropagation?

Microcircuits could descend the gradient of a function they can evaluate locally.

It is perfectly possible to build neural networks with other functions besides gradient descent.

I agree. I'm saying we don't know how biological neural networks are trained, which is a pretty important part of any learning construct, so we can't really know how actually similar artificial neural nets are.

What other methods are there for training a neural network? Even blind evolutionary algorithms (assuming each generation is initialized to be somewhat 'near' or 'similar' to its progenitors) are effectively descending a gradient.

To my knowledge they explore solution space but not through a gradient descent but by probing.

How can you perform a GD on non continuous function? Genetic algorithms and evolutionary methods can explore disjoint domains and functions hard from analytical pov.

Just because they can be used for gradient descend doesn't mean they use it.

Reply on news.ycombinator.com