Hacker News Reader

Comment by gus_massa | Hacker News Reader

The official name is https://en.wikipedia.org/wiki/Chinese_room

The opinions are exactly the same than about LLM.

And the counter argument is also exactly the same. Imagine you take one neuron from a brain and replace it with an artificial piece of electronics (e.g. some transistors) that only generates specific outputs based on inputs, exactly like the neuron does. Now replace another neuron. And another. Eventually, you will have the entire brain replaced with a huge set of fundamentally super simple transistors. I.e. a computer. If you believe that consciousness or the ability to think disappears somewhere during this process, then you are essentially believing in some religious meta-physics or soul-like component in our brains that can not be measured. But if it can not be measured, it fundamentally can not affect you in any way. So it doesn't matter for the experiment in the end, because the outcome would be exactly the same. The only reason you might think that you are conscious and the computer is not is because you believe so. But to an outsider observer, belief is all it is. Basically religion.

But you are now arguing against a strawman, namely, "it is not possible to construct a computer that thinks".

The argument that was actually made was "LLMs do not think".

A: X, because Y

B: But Y would also imply Z

C: A was never arguing for Z! This is a strawman!

"LLMs cannot think like brains" does not imply "no computer it will ever be possible to construct could think like a brain".

“LLMs cannot think like brains” is “X”.

That appears to be your own assumptions coming into play.

Everything I've seen says "LLMs cannot think like brains" is not dependent on an argument that "no computer can think like a brain", but rather on an understanding of just what LLMs are—and what they are not.

It seems like the brain "just" being a giant number of neurons is an assumption. As I understand it's still an area of active research, for example the role of glial cells. The complete function may or may not be pen and paper-able.

> The complete function may or may not be pen and paper-able.

Would you mind expanding on this? At a base read, it seems you implying magic exists.

It could well be the case that the brain can be simulated, but presently we don't know exactly what variables/components must be simulated. Does ongoing neuroplasticity for example need to be a component of simulation? Is there some as of yet unknown causal mechanisms or interactions that may be essential?

None of those examples cannot be done on pen and paper or otherwise simulated with a different medium, though.

AFAICT, your comment above would need some mechanism that is physically impossible and incalculable to make the argument, and then somehow have that happen in a human brain despite being physically impossible and incalculable.

There are indeed many people trying to justify this magical thinking by seeking something, anything in the brain that is out of the ordinary. They've been unsuccessful so far.

Penrose comes to mind, he will die on the hill that the brain involves quantum computations somehow, to explain his dualist position of "the soul being the entity responsible for deciding how the quantum states within the brain collapse, hence somehow controlling the body" (I am grossly simplifying). But even if that was the case, if the brain did involve quantum computations, those are still, well, computable. They just involve some amount of randomness, but so what? To continue with grandparent's experiment, you'd have to replace biological neurons with tiny quantum computer neurons instead, but the gist is the same.

You wouldn't even need quantum computer neurons. We can simulate quantum nature on normal circuits, albeit not very efficiently. But for the experiment this wouldn't matter. The only important thing would be that you can measure it, which in turn would allow you to replicate it in some non-human circuit. And if you fundamentally can't measure this aspect for some weird reason, you will once again reach the same conclusion as above.

You can simulate it, but you usually use PRNG to decide how your simulated wave function "collapses". So in the spirit of the original thought experiment, I felt it more adequate to replace the quantum part (if it even exists) by another actually quantum part. But indeed, using fake quantum shouldn't change a thing.

> component in our brains that can not be measured.

"Can not be measured", probably not. "We don't know how to measure", almost certainly.

I am capable of belief, and I've seen no evidence that the computer is. It's also possible that I'm the only person that is conscious. It's even possible that you are!

I don’t understand why people say the Chinese Room thing would prove LLMs don’t think, to me it’s obvious that the person doesn’t understand Chinese but the process does, similarly the CPU itself doesn’t understand the concepts an LLM can work with but the LLM itself does, or a neuron doesn’t understand concepts but the entire structure of your brain does

The concept of understanding emerges on a higher level from the way the neurons (biological or virtual) are connected, or the way the instructions being followed by the human in the Chinese room process the information

But really this is a philosophical/definitional thing about what you call “thinking”

Edit: I see my take on this is listed on the page as the “System reply”

If 100 top-notch philosophers disagree with you, that means you get 100 citations from top-notch philosophers. :-P

Check out eg Dennett.... or ... his opionions about Searle; Have fun with eg... this:

"By Searle’s own count, there are over a hundred published attacks on it. He can count them, but I guess he can’t read them, for in all those years he has never to my knowledge responded in detail to the dozens of devastating criticisms they contain;"

https://www.nybooks.com/articles/1995/12/21/the-mystery-of-c...

Reply on news.ycombinator.com