Hacker News Reader

Comment by IIAOPSW | Hacker News Reader

I honestly think we will one day look back and see this as taking a hammer to a screw. Sure it will work, but its a horribly brute-force way of doing it. The brain almost certainly processes language in a more elegant way that, once understood, certainly won't take up such raw computational power.

The brainwaves associated with a wakeful state (and thus generating and parsing language) happen around 40 Hz. Our machines are running in the GHz region. We are missing some fundamental insight.

At the risk of ending up front and center in the hall of prediction hubris, here is my bold prediction: By 2040 it will be possible to do what these TPU's are doing today on your consumer grade computing device, but it won't be because of a dramatic increase in the computing resources available to such devices.

View on HN

I agree that this stuff will be orders of magnitude more efficient once we figure it out. But I think brute forcing it first and optimizing later is a fine way to do it. Maybe even the only way to make progress on a problem like this. And even in a hypothetical future where you can run human level intelligence on your smartphone, supercomputers will be that much more intelligent still, so it's not like building them will be pointless.

I mean shoot, if you brute force it enough you can ask the machines to optimize for you at some point.

I'll tip my hand further. I think the current approach to artificial neural networks is inherently flawed because it simplifies away pertinent features like firing rates, propagation delay, and standing waves (aka brain waves). I also think trying to get AI to "understand" the irl world when it doesn't live in it and we haven't codified our own natural language is too far a leap. We might have more success making an AI that is conversational in a form of newspeak dedicated to a specific problem domain. Eg an intelligence with the body of a brokerage account that only knows how to speak about money (thus rendering most of humanity redundant (zing!)). Here's something I wrote elsewhere about the accidental path to machine intelligence.

"You are whatever you get in return when you ask yourself who you are".

Literally, if you trace the referent objects, "you" refers to the same entity as the voice which responds to your query. But also literally, "you" are the standing waves of feedback loops in the brain which represent thoughts.

Rather subtly, I've given an understanding of consciousness which is entirely defined and constrained within a communicative system of sending and receiving messages. The underlying language within which the consciousness exists may have a very minimal grounding in some physical reality.

(I'm truncating a ton of justification and background to avoid a deep rabbit hole and get to the point).

Here is a highly speculative example of how consciousness-like intelligence could emerge accidentally in the form of trading bots. It starts with two ingredients: the shared reality of the order book and independent trading bots with the objective of "make more money".

At first the algorithms are dumb curve optimizers. They see whatever patterns they can find in the order books and correlate them with outcomes, reacting appropriately by placing their own orders.

After a while, as the data sets and internal models grow, the algorithms are implicitly calculating the reactions of other trading bots in their extrapolation.

Soon after that, the orders being placed in the order books are effectively messages intended for other trading bots, hoping to illicit a certain reaction. The order book effectively becomes a language grounded in an economic reality, filled with offers that are communicative but not necessarily intended to execute.

The emergent language of the order book grows in sophistication to the point where the bots are talking about past instances and hypothetical instances of things said in orderbook speak. This all happens right under our nose in perhaps non-obvious ways, such as in the exact number of cents on a bid/ask price. Other times it looks like our bots have reinvented "painting the tape" and other forms of financial communication we've deemed "market manipulation". We're proud of our silicon brain child for figuring out how to do that on its own. They grow up so fast.

Eventually, in the process of optimization, the trading bots internal model of its "order book reality" gets to a point of sophistication where it has to model other bots modelling itself modelling other bots modelling itself...to whatever layer of recursive depth it can marshal. Thus the feedback loop effectively closes and something akin to circular loop brain waves emerge. By this point no one really understands why the trading bots make the trades they do. I can no longer tell you what a trading bot is in simple terms like "its a dumb curve optimizer trying to maximize money". Rather, it can only be understood as a form of consciousness which has emerged:

"The trading bot is whatever the trading bot gets in return when it simulates placing an order asking what it is."

Whenever we try to define consciousness, we get into useless quasi-philosophical discussions like this. Philosophers don't know what consciousness is and neither does anyone else.

Working with your definition, you've basically just described backpropagation in sufficiently deep neural nets. A feature of Artificial Neural Nets, which, as you say, probably oversimplifies brain functions.

"The underlying language within which the consciousness exists may have a very minimal grounding in some physical reality."

Your usage of physical reality is interesting. Is this the "free will is real" ala true randomness exists argument again? Hope we aren't moving backwards into the arms of religion here.

Technically any loop can be unrolled into a sufficiently deep linear sequence. Technically it can be done, but good luck conceiving of solutions to routine problems without while loops, for loops or recursion. It can be done, but you have to think harder, write much longer / more repetitive code, and looking at the sort of code that doesn't contain insights about the problem.

So, back-propagation with sufficiently deep neural networks. Technically you could use it. You could throw a huge amount of silicon and brain power at any problem and eventually hammer that screw right into the wall. Or you could try slightly more realistic models of neurons and hope to find disproportionate increases in the abilities over the previous model. I think its already clear which one I'm in favor of.

Just to make this extremely explicit, the thing I think is missing from artificial neural networks is harmonic waves. There's a body of evidence that representation of thought is done with brain waves, not the states of individual neurons[1]. When you move to a wave view of neural networks a handful of very sophisticated operations emerge naturally such as autocorrelation (effectively a time windowed fourier transform). Sure you could program the fourier transform, or even worse get an optimizer to implicitly learn it after some outrageous number of man hours, but in this analog wave view of brain activity we get it with structures so simple they could have happened by accident. I'm being extremely literal when I say "you are what you get in return when you ask yourself". The voice in your head is literal the echos of the question bouncing around in your skull (albeit electro-chemically rather than acoustically).

I am arguing that consciousness, that is to say the train of thought in your head, is definitionally what happens when the conversational abilities of understanding utterances and forming responses get fed into each other. Consciousness is nothing more than talking to someone who happens to be yourself. Maybe my definition doesn't have universal acceptance, but it at least gives a meaningful concrete answer to what is meant by consciousness.

You've far and away missed the point on "grounding in reality". It has nothing to do with randomness or free will or religion. I didn't hint at anything of the sort.

Someone once said something profound to me. "In a programming language, no matter how much complexity or abstraction there is in a command, everything eventually resolves down to instructions to physically move some electric charges at a physical location in memory." Something similar applies to natural languages. Every sentence and though eventually resolves down to representing physical and tangible things in our reality. When you try to trace through the dependency tree of the dictionary, you eventually reach words which can't be broken into simpler parts. Those are the words that "ground" the language in our physical reality, representing objects in the outside 1 to 1. Every language, be it natural or programming or something else, has some form of grounding. Language has to be about something. But the underlying thing its about can very simple. It can be as simple as an order book.

To any conscious entities that emerged in such an accidental medium, the order book is their reality. They wouldn't know of or be equipped to reason about any other form of existence. Their form of existence is no better nor worse than any other consciousness grounded in the reality of any other language. It doesn't matter much what the underlying objects of the problem domain are. I picked trading bots as my example, but I could have picked any other domain where agents 1) share the same playing field 2) have some competing interests to optimize and 3) could use objects of the domain for signaling purposes (ideally at low cost).

[1]https://www.nature.com/articles/ncomms10340

But brains are trained from millions of generations across the lifetime of the earth (or from when life started, if you ignore the environment and context needed to start life). You are conflating the training with the inference.

at least in the visual cortex, synaptic weights are not genetically encoded in humans. they are in drosophila though

Source? I am skeptical that we even can know that because so much can be implicitly encoded.

I do not have a single source, that result is represented across thousands of drosophila papers. There was a famous paper were they showed that fruit flies bred in the dark for many generations showed no significant optic lobe tuning differences to wild type flies, but I don't remember who did that.

The extreme plasticity of the human visual cortex is widely documented (studies in blind people, etc).

> The extreme plasticity of the human visual cortex is widely documented (studies in blind people, etc).

I think there is a big gap between the two claims -

1. Human visual cortex is very plastic

2. There is no inductive bias in the visual part of the brain that has been "trained" by millions of years of evolution.

I trust you are correct on the former, but the latter does not seem to follow from the former.

Certainly there is some inductive bias. Neuroanatomy is broadly similar human to human which constraints connectivity. There is an energetic penalty for excess wiring - evolution drives neuron placement to minimize wiring length (eg topographic layout of layers in cortex), which further constraints connectivity. In general, there is a huge body of evidence supporting the hypothesis that organisms are highly tuned to their environments.

My point is that in contrast to fruit flies, humans (well, mammals) raised in environments with non-natural scene statistics show significant differences in coding in the visual cortex. Here are some random references I dredged up:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210638/ https://www.sciencedirect.com/science/article/pii/S096098222...

It's not hard to see why this might be selected for in mammals. The historic range of D. melanogaster is just sub-saharan Africa, whereas the historic range of humans is huge, and we are primarily visual organisms. So it would be advantageous to have different tuning if you lived in the desert vs the forest. Of course you have to look at the range and divergence times of the full phylogeny but I am getting distracted from my actual research. It's a very interesting line of thinking though. I was quite surprised to learn things were hardcoded in flies when I joined a fly vision lab.

https://academic.oup.com/gbe/article/11/3/844/5304659

> The brainwaves associated with a wakeful state (and thus generating and parsing language) happen around 40 Hz. Our machines are running in the GHz region.

I don't disagree with the larger point, but each neuron is like a small independent analog computer, so you have tens of billions of (specialized) cores running at 40Hz, vs thousands of cores at ~GHz.

I'm sure we will end up with better solutions, but it's a lot easier to study a thing when you have an introspectable working model. I expect that many AI patterns will be that we'll build it, then optimize it and learn how and why it works, taking the art and science into engineering piece by piece.

Human brain has very likely 10^15 synapses so these models are still behind at least 3 orders of magnitude. To put in perspective, if you want to simulate “network” of human brain, you probably need an entire data center worth of GPUs.

That is assuming you restrict yourself to a radical simplified model of each neuron. The actual biochemical processes going on in a single synaptic density go way beyond something that could be represented by a scalar weight that assemble into a linear transformation. Instead we are talking about 10^15 dynamical systems. This then still neglects most of the time /spatial extended nature of neurons, the fact that calcium stores in the endoplasmatic reticulum play a role etc. Some of this might be only relevant to learning, but non-linear dendritic processing is believed to be relevant for example for inference aswell.

The brain is massively parallel (itself an understatement). Computers are not.

Reply on news.ycombinator.com