Hacker News Reader

Comment by fwlr | Hacker News Reader

This is layman speculation, but… The human brain is notoriously efficient; neural networks are modeled on the brain so they’re probably doing something like what the brain does; the brain uses electrical signals and these optical transformers use light signals; Maxwell’s work unifies electricity and light. It all seems to add up - the general process is “take advantage of an energy field to cheaply perform calculations on a mass scale”, brains build a biological/chemical structure to use the electrical field, optical transformers will build a silicon structure to use the light field.

View on HN

My understanding is that modern neural networks aren’t much like the brain, but machine learning researchers like the way it sounds to suggest that they are.

NNs are inspired by the brain, it's wrong to suggest they're not and also wrong to suggest brains are _merely_ NNs.

Activation functions in NN map to action potentials/postsynaptic potentials, weights map to synaptic strengths/connection patterns between neurons, dropout to synaptic pruning/apoptosis, convolutional layers from the receptive fields of our visual cortex.

Certainly our brains are more complicated. Temporal spike timing, dendritic signals, synaptic plasticity, compute of synapse themselves, lots more I can't recall, none of these concepts are currently modelled in our NN architectures.

NNs start from the simple premise of interconnected neurons. As the field develops and we find what works or what doesn't, perhaps we will take more bits of inspiration from the brain.

I'll add that there's no evidence that brains do backpropagation

And there’s research coming out that suggests that forward propagation can implement the same thing but comparatively cheaper without much loss in accuracy and is likely the next step. I don’t think we really know enough to conclude things either way and we don’t know if brains never did back propagation or evolved it away.

Expanding on this history for readers here who are just catching up on this stuff, modern feedforward NNs descended from the Perceptron paper by Rosenblatt and friends from 1957. This was the first paper that really tried to position "learning machines" based on accumulated stimulus activations, and it built off of a previous paper from 1943, "A Logical Calculus of Ideas Immanent in Nervous Activity." So yes it's true that the field borrowed ideas from the brain and it was positioned that way from the start, though Perceptrons totally weren't a very similar implementation to what Nature herself designs...

A Perceptron model is just a dot product: one multiply+add, summed to one output scalar value. Modern nomenclature might consider Perceptrons to just be "one channel of a single layer," or like "one fully-connected layer with a single output." In fact, that's where the old-fashioned name for neural networks came from: "Multi-layer Perceptrons (MLPs)" are a bunch of these arranged on top of each other. The older 1943 work showed that in some sense lots of possible arrangements are roughly equivalent to each other so it made sense for Rosenblatt and friends to start from the simplest model with the understanding that the field would build from there. (Our feedforward model ancestry is just one branch of the tree rooted in this work; there's a completely separate geneology for other possible arrangements of modeling components, like recurrent Boltzmann machines that simply haven't been so well studied/understood... :-)

The generality of Perceptrons made room for a lot of interpretive flexibility. Right from the start, news reporters saw the theory, attended academic conferences, and started writing about it. The NYTimes had an article in 1958 that spoke of the Perceptron as "the embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence."

In fact there was a very famous kerfuffle between Rosenblatt (Perceptron authors) and Minsky & Papert (another AI researchers). The latter two published a spicy book in 1969 saying that a single Perceptron wasn't general enough because linear models can't even learn an XOR function. Everyone who read this book misinterpreted it as a takedown of Perceptrons; this book inadvertantly dried up interest, choked funding across the board, and ultimately caused the first AI winter which didn't get resolved until well into the '80s when these ideas started to be revitalized. You can read about the spilled tea here, it's quite fascinating: https://doi.org/10.1177%2F030631296026003005

I think the cause of the first AI winter had more to do with some of the early researchers suggesting they'd have solutions to problems within a few months to years that are just now becoming tractable. In retrospect it doesn't seem like they could've gone much faster than they did. New ideas mattered but increased computing speeds probably mattered more. It's not like we would have been able to get dramatically better results than those researchers did by taking what we know now and running it on the same hardware they had back then.

ML researchers I've spoken to strongly dislike the way people suggest similarities between biologically-inspired systems and neural networks. The earliest papers did use our understanding of the neuron as inspiration for some of the ideas, but these approaches have far more similarities with statistical analysis than they do to neuroscience. At the end of the day, NNs are just nonlinear function approximation, and if I had my way we would have called it that from the start.

Choosing positioning like "NNs are similar to the brain" or even calling the field "machine learning" makes it harder to speak objectively about the research we perform, makes it hard to understand the sort of explanatory power and limitations of the models we fit, and makes it harder for users to understand how the system works. It's like how quantum mechanics researchers have to deal with readers who misunderstand what linear observable operators are and conflate it with their own ideas about conscious observers or use it as evidence of God's presence or whatever.

This! I was listening to a radio interview with a famous NN researcher and everything was going great until the interviewer asked him a question that only a neurologist could answer, and he kept talking in exactly the same way, answering as though he was a neurologist! I was embarrassed for him. It seemed to me that the correct and useful response would have been something like what you just said.

Reply on news.ycombinator.com