Hacker News Reader

Comment by js8 | Hacker News Reader

I think there are two important factors to this.

One is that the operating frequency of the human brain is much lower. That results in much smaller energy consumption.

The other is that current neural networks are probably very inefficient algorithms. They are fundamentally based on linear algebra but there is some evidence that the non-linearity is the key.

My feeling is also that brain extensively uses probabilistic algorithms with hash encoding, conceptually similar to bloom filters, count-min, etc.

View on HN

My theory is that, instead, the brain is a super-duper complex neural network with some differences about how the update of the weights happen :) Let me explain:

Albeit true that our brain is much slower than a modern CPU in terms of ops/s, every neuron is connected to up to 10000 other neurons[0]. And we have something around 100 billion neurons in our brains. This means that the number of synapses is between 100 and 1000 trillions.

If we assume that every synapse can be considered like a "weight" in a neural network, we have a model that is much much bigger than anything developed to date. Moreover, the topology of said neural network is not a simple stack of layers, but much more complex like a super intricated residual network.

Finally, the brain is not using backpropagation to update the mentioned weights/synapses' strength, but instead is based on how often said synapses fire.

[0]: http://www.human-memory.net/brain_neurons.html

People also gloss over the fact that there are multiple neurotransmitters involved.

> If we assume that every synapse can be considered like a "weight" in a neural network, we have a model that is much much bigger than anything developed to date.

This is a very bad assumption. Each synapse is a highly complex chemical computer, and its "weight" with regard to its contribution to the postsynaptic neuron's activation at any point in time depends on a multitude of factors.

Those factors include dozens of interacting metabolic pathways, the interactions of various neurotransmitter systems which each have distinct behavior, and things like where the individual synapses sit relative to one another on the dendritic arbor. There's even evidence that at least in some neurons the activation pattern may influence RNA production in the neuron, creating a kind of recording which influences the behavior of the cell.

Current ANN models should be considered a very low dimensional approximation of the real thing.

> My theory is that, instead, the brain is a super-duper complex neural network with some differences about how the update of the weights happen

Neural networks were not meant to be accurate analogues to the actual operation of the neurons in the brain. They were known to be poor matches for the actual biology, even back in 1959. The concept of the multilayer neural network was actually inspired from our knowledge of the visual cortex, rather than the lower-level fundamentals of the brain. The brain is most certainly not a neural network, as the term is understood in machine learning.

> The brain is most certainly not a neural network, as the term is understood in machine learning.

The brain is most certainly a neural network. It is literally a network of neurons. But you're right insofar that artificial neural networks are not very similar to the brain.

That is exactly what the parent comment said.

I was objecting to the statement: "The brain is most certainly not a neural network" - as if to say that the types of artificial neural networks employed in machine learning are somehow the central definition of what a neural network is. I would be happy with "the brain is not an ANN", or "The brain is very different from the types of neural networks used in Machine Learning", but to say the brain is anything other than a neural network is just plain wrong.

That's not what they said. They said, "The brain is most certainly not a neural network, as the term is understood in machine learning.".

I understand what they said. That statement reads: "Machine learning has a concept of a neural network, and the brain is not that". A correct statement would be: "The field of machine learning makes use of a computing system called an artificial neural network, which is loosely based on the structure and function of a biological neural network."

The statement in question would be like saying: "Italians most certainly do not make real coffee, as the term is understood at Starbucks." There's a technically correct reading of that sentence, but the implication is offensive to the subject matter.

The brain has other differences that are probably a major influences on it's efficiency;

1) It's incredibly parallel, few operations happen linearly. Even modern GPUs don't really match that.

2) It doesn't run everything all the time. Parts of the brain not in use are shut down. Modern computers do this to some extend but CPU cores don't switch off parts of the ALU not in use. The brain has much better energy saving options, only 10-30% of the brain are active at a given time.

Building on 2- this is likely not a result of saving energy but rather hyper-specialization in specific regions of the brain. Like if modern CPUs had 10-20 cores, each tuned to handle a specific task. The result is energy efficiency, but in contrast to current energy saving options, the brain can't simply "speed up" and use more power and activate more areas of the brain.

> It's incredibly parallel, few operations happen linearly. Even modern GPUs don't really match that.

Modern GPUs aren't even close: neither in terms of degree or in kind. A GPU does a lot of linear operations in parallel, but in the brain all the "compute units" are interacting with each-other in real time.

> The other is that current neural networks are probably very inefficient algorithms. They are fundamentally based on linear algebra but there is some evidence that the non-linearity is the key.

Deep neural networks, though they use linear algebra, are definitely non linear, as a result of both their architecture and their use of non linear activation functions.

Their non-linearity is part of the reason they can classify phenomena that linear techniques like simple regression cannot.

What about the idea that the brain is analog?

The brain is analog.

It's digital and analog. At the level of the action potential, the brain certainly does seem to operate with discreet units of information. Everywhere else though is certainly analog.

So if a neuron fires or not (I guess that's what you mean by action potential) can simply be translated to 1 or 0 hence it's digital?

Basically yes. That's only one small piece of the information processing of the brain, but it would in principal be possible to represent the "firing state" of a biological neural network in binary terms.

Reply on news.ycombinator.com