Hacker News Reader

Comment by colinmhayes | Hacker News Reader

AGI is only inevitable if you subscribe to materialism. Basically it comes down to whether we are purely material beings or whether there are non-material aspects to our mind. There are more than a couple non religious arguments for dualism, many are relatively recent, as dualism is enjoying a bit of a comeback in philosophical circles. The Chinese room and the red room a.k.a.Mary's room are simple enough thought experiments that can help you grok why dualism might be true.

That being said, the most recent philpeople survey has 52% accepting materialism compared to 32% accepting dualism with the rest undecided, so academic philosophy does generally lean toward AGI. https://survey2020.philpeople.org/survey/results/all

View on HN

The Chinese room argument is a parlour trick that uses scale as a distraction. It posits a person in a room manipulating symbols to produce intelligent seeming outputs. It says, see, it’s absurd to think a person in a room with a stack of symbols could emulate intelligence.

But let’s say the room contains many billions of people, it is the size of a planet, and it contains racks of many trillions of symbols, and it spends millions or billions of years to produce an output. That’s more like the scale of a sophisticated computer system, or a brain.

Does that sound much like a man in a room with some symbols? No. Does it sound like that could do complex calculations and produce sophisticated and perhaps even intelligent outputs? Well, given enough time and scale, yes why not?

The Chinese room is pure misdirection and it amazes me anyone falls for it. There’s really no actual argument there.

I mean I agree, but a third of philosophers subscribe to dualism, so I wouldn't be so quick to write the whole thing off.

Dual meaning two, right? Material and non material. As far as I can tell not one of them can describe what the second non material thing in dualism is, or what any of its properties are. Even they seem to have no idea what they actually believe.

They believe it's not possible to explain human consciousness as a purely physical concept. I don't see why they need to explain what it is that isn't material to make that claim.

I just want a coherent explanation of their belief. They’re the ones positing something non material, not me. I just what you know what it is they mean by that. Absent that, I don’t see how they can claim their belief is coherent.

I’m not especially well versed, but as I understand it they believe we don’t know the essence of the non material part and in fact it being unknowable is sort of the point.

This sort of reminds me of the people who claim the Big Bang is incoherent because we don’t know what came before it. Why is I don’t know not good enough?

I suppose that’s fair to a point. It’s just disappointing.

The thing is this non material whatever can’t have properties. If it stores information, has state and has consistent behaviour then really it’s just a form of material (in the philosophical sense that it’s a part of the world), so it can’t have any of those. At which point, how does it even interact with the world at all? I mean if it influences the brain, then that mechanism of influence makes it part of the world, right? It makes it material.

That’s where I find the idea incoherent. I don’t see how it can be both immaterial but also have material effects. Doing so makes it material, I think by definition. It means it must be part of material reality in that sense. Or at least I’d like to hear an argument why it doesn’t.

What is the material evidence for numbers? What is the material evidence for logic? What is the material evidence for persistence of identity over time? What is the material evidence for math? What is the material evidence for categories? What is the material evidence for grammar? What is the material evidence for a mayor (not the mayor qua person, but the mayor qua office)? What is the material evidence for marriage? According to empirical evidence, which knife is the best knife? There is no "best" without purpose. And materialism has nothing to say about purposes. Empiricism can only work when an agent in the world has a specific purpose. There are vast swathes of human experience that don't fit tidily in the box of materialism or empiricism.

Those things are behaviours and information that can be encoded in matter. If the non material thing in dualism is just emergent behaviour, then it’s not adding anything beyond what we already have in materialism.

Numbers aren't emergent behavior. Numbers are a metaphysical category. Categories are metaphysical. There's no empirical evidence for order vs. randomness. The concept of "order" doesn't make sense empirically. There has to be a separate, metaphysical value structure that determines whether the data aligns to the value structure or not. Just because you haven't been clear in your thinking doesn't mean metaphysics aren't real or essential to dealing with the physical world.

I accept that. Metaphysics is real in a useful way, but I think that metaphysics itself is an emergent behaviour. Emergent behaviours are behaviours of mater, and matter is real, therefore emergent behaviours are real.

"Emergent behavior" is handwavium materialists use to avoid cognitive dissonance.

We observe them, categorise them, and engineer them all the time.

If the "non-material" is required to explain material phenomena, then it must be causally related to the "material"...but at that point what exactly makes it "non-material" at all?

"Non-material" just sounds like a bad label for material things we don't understand.

We don't understand how our cognition or consciousness works, but it seems silly to assume that because we don't understand we fundamentally can't understand.

People who push the mysterious interpretation of dualism are just trying to find a place for the divine that is separate from the material but in doing so they baselessly seek to circumscribe our capacity to understand.

Imagine an Apple. Is that Apple material?

Those neurons firing while imagining are material, yes.

But where is the apple? That is, where is the emergent phenomenon of the apple?

The emergent phenomenon is a pattern in my brain, just as a running a computer program is an emergent phenomenon in a computer.

If the extra thing dualism adds is just behaviours of matter, how is that different from materialism?

Who is reading the Apple?

I'm not sure what you mean by reading.

We say that we can experience thinking about an Apple, or imagining an Apple. I see no reason why a computer, or other physically implemented AGI system, could not do that. I suspect the act of imagination is just generating, processing and transforming a computational model abstracting the thing being imagined.

I believe brains are physical objects, so therefore physical objects can imagine Apples.

The plane of imagination isn’t physical, else you could touch it.

Sure you can, its done currently for Deep Brain Stimulation. What you're proposing is "The plane of Quake isn't physical", which is nonsense.

I can make the computer imagine Quake for me, then fiddle with its plane of imagination for some sweet wallhacks.

If you create a device that can insert objects into the plane of someone’s imagination, that would be awesome :)

That's a side effect of DBI today. It's a random and crude method today, but arguing that we we'll hit some ineffable wall that will prevent more fine-grained control is.. well let's just say that the gaps for gods grow ever smaller.

I don't know what a plane is in this context.

You can't touch fourier transforms either, but a human brain or a microchip can compute them.

Who is it that fetched this comment from the internet for you?

Using vaguely-defined words to support a deist position is a time-honored tradition, but isn't particularly interesting or convincing.

There isn't a platonic ideal "Apple" if that's what you mean by emergent phenomenon.

A collection of neurons can build a model of the world that includes its experience of apples, and from that, dedicate some neurons to representing a particular instance of an apple. This model isn't the reality of "Apples", though, and is physically located in the brain.

“ There isn't a platonic ideal "Apple" if that's what you mean by emergent phenomenon.”

Sure there is, that’s what DeepMind showed us with how to find cats in images :)

That's exactly my point. DeepMind has an idea of what a cat is based on its experiences, just as you or I do. Each of our models are woefully incomplete, based on very limited sensory information. These models all disagree with each other and reality to various degrees.

There exist many things many humans have lumped together under a single label such as "cat". Those categories are all wrong, but sometimes they're useful. Machines can also get in on the fun, just as well as humans, as you point out. There's no magic there, humans aren't special.

The humans aren’t special bit really comes down to whether you believe in free will (which by any meaningful definition is quite special).

Free will is another one of those things that people love to trot out because it's so ill-defined. To cut through all the crap though, it's very simple: "free will" === "unpredictable behavior". This inherently means that it's observer-dependent.

This has the benefit that it empirically fits how people think about it. Nobody thinks a rock has free will. Some people think animals have free will. Lots of people think humans have free will. This is everybody trying to smush a vague concept into the very simple, intuitive definition above.

Which is all to say that free will is about as relevant to any conversation as say astrology is: not one bit.

As much as a cloud of atoms doth protest that free will is irrelevant, reality has a way of not caring :)

Funny, I'd say reality has a way of existing despite all of the comforting woo people like to make up about it.

can't describe the system from inside the system, boss

Seems like something someone believes out of hope right?

That there is something more than this. That it isn't just over. Something to keep people from giving up.

Edit: hope it didn't come off like I was talking down to people who believe what they believe. It's your life, it's your after life. Stay safe yall.

Dualism is simply an ivory tower God of the gaps. To these sophists, anything we can't explain now can be decomposed into a physical and metaphysical part.

> It says, see, it’s absurd to think a person in a room with a stack of symbols could emulate intelligence.

That's not at all what it says. It says:

1. Assume a computer program can pass a turing test

2. Convert the computer program into a list of steps that a human in a room can follow

3. Put a human in the room. Pass him a slip of paper with Chinese writing. Have him follow the steps to generate a response.

4. Now consider, once you've done this, does the human in the room following those steps understand Chinese?

Searle's answer is obviously not, from which he deduces that the computer program doesn't understand Chinese either.

Right, but the person in the room isn’t the computer program. He’s not even the whole computer. He’s just a component of the system. It’s the system that we should consider as understanding Chinese, not just a piece of it. He’s following a set of instructions so he doesn’t even have all of those in mind at once either.

By abstracting the system as a man in a room were distracted to think we’re considering the system when we’re not. If the man in the room is one of billions of men, at massive scale which is a more realistic model, that becomes obvious.

Yes, one response to the thought experiment is "the whole system understands Chinese". I don't find that response convincing. Do you really think a book with instructions in a room plus a man to carry them out understands Chinese?

The thought experiment hinges on the concept of understanding. If you want to argue against Searle, and you can, your best bet is to attack that concept, to argue that "understanding" is an illusion or that it's nonsensical. This is what philosophers like Daniel Dennet have done.

> Do you really think a book with instructions in a room plus a man to carry them out understands Chinese?

Again with the scale misdirection. Are you really sure a room the size of Jupiter with a rule book the size of many libraries, full of trillions of people following the rules, shuffling quadrillions of symbols for millions of years definitely can’t ever understand Chinese? Not in any sense?

You're bringing up scale, not me. The book can contain 10 billion lines of instructions. That's a lot of "scale".

The thought experiment is about the concept of understanding/intelligence/consciousness/whatever. It has nothing to do with scale.

If it has nothing to do with scale, then why doesn’t Serle us a realistic example of a vast room, why a book not a library, why a man not an army?

I’m bringing up scale because I’m pointing out how it’s being used.

If you insert the adjective "vast" before room, then you get The Vast Chinese Room thought experiment and nothing changes.

A lot of people find it credible that a hugely complex fast computer system could be intelligent. Many of them seem to find it convincing that a man in a room with a book can’t “understand Chinese” when he personally doesn’t. But the only difference between them is scale.

> A lot of people find it credible that a hugely complex fast computer system could be intelligent.

But not John Searle...so it seems clear that scale has nothing to do with the thought experiment. For Searle, a vast room or an army of people to carry out instructions make no difference.

> Many of them seem to find it convincing that a man in a room with a book can’t “understand Chinese” when he personally doesn’t

The people who read the thought experiment this way don't understand it. In the thought experiment neither the computer program (no matter how powerful the computer is) nor the man and the room understand Chinese.

> But the only difference between them is scale.

No, the point of putting a man in the room is that we all agree that a human is capable of understanding. From his vantage point in the room, he can see that the program works by executing instructions without any understanding. All the room does is demystify a hypothetical computer program that can pass a turing test.

Searle is not arguing that a machine with understanding/consciousness is impossible. He's arguing against behavioralism ("if it can pass a turing test it understands"). He take the psychological phenomenon of "understanding" (something we're all familiar with) seriously, he thinks it represents a real thing, and that a computer that can pass a turing test doesn't necessarily have it.

I don’t see how that’s different from a neurologist examining a human brain. They can see it’s just cytoplasm and electro chemicals oozing around and firing off signals. None of that looks like it understands anything, any more than the symbols moving around in the Chinese room. They can no more point to the place in the brain that understands things than the guy in the Chinese room holding the book can.

The difference is that we all have the subjective experience of understanding. When we have a conversation, we experience other people's words as sounds in our heads laden with meaning. We "understand" each word in the sentence, how the words fit together, what they refer to, and so on. We're all familiar with this experience, right? We're doing it right now through the medium of writing.

Searle takes that experience seriously. For him it's real and, in order to have that experience, you need machinery somewhat similiar to what we have. He can't bring himself to believe that a list of instructions and a person dumbly executing them have that sensation of understanding even if they produce the correct output.

I take that experience seriously too. I don't see why a sufficiently advanced artificial system as complex, capable and sophisticated as a human brain could not also have experiences. I don't think he has any good reason to believe otherwise, or any good argument against it. Nothing he says about the mechanisms of the Chinese room can't also be said about the mundane physical mechanisms of neurology.

> I don't see why a sufficiently advanced artificial system as complex, capable and sophisticated as a human brain could not also have experiences.

Neither does Searle. He says that humans are such machines. Per Searle, if you want to build a system that has subjective experiences of consciousness and understanding, it would have to have physical parts that correspond to some degree to the physical parts that give rise to our subjective experiences (or an animal's subjective experiences). It couldn't be a list of instructions to be executed.

I don't know if Searle is right about this but I find myself unable to dismiss his argument as obviously wrong.

You don't think it's obvious that actual computers and robots have physical parts? I'm sorry, I'm at a loss.

> The people who read the thought experiment this way don't understand it. In the thought experiment neither the computer program (no matter how powerful the computer is) nor the man and the room understand Chinese.

Who says the computer doesn't understand Chinese, though? In this case "the computer" is the system. The program, being the code, isn't a process. The man is acting as a mechanical piece of hardware. But the computer running the program is an active process with state.

I heard a lecture by Searle on this. He mentioned the idea that "the room understands Chinese" and simply dismissed it as absurd without considering it. This is exactly the problem with his argument. Well, that and the casual conflation of the man with the computer as if the CPU hardware is the system.

Right, exactly, it is absurd to think one man in a room with some symbols can generate understanding. But only because of scale, because we imagine a regular sized room, a normal sized book and a table with symbols on it. But it’s absurd because that system is incredibly simple, not due to any actual argument from fundamentals.

I think something like OpenAI CLIP + Gpt3 "understands" English at a level comparable to a 7yo.

It can respond to questions and explain its answers. Sometimes it's wrong, but so is a 7yo and you can usually understand why.

Is it intelligent? Maybe it is.

> Do you really think a book with instructions in a room plus a man to carry them out understands Chinese?

Do you really think a blob of goo with static sputtering through it, designed by no one, accidentally mutated from pond slime, understands Chinese? I actually find your proposition easier to believe, but here we are.

This seems like saying since a person's individual neurons can't understand Chinese, then a person can't actually understand Chinese either.

We know how the room works--not the brain.

...Can we break it down and replicate it?

Can my sense of self be quantified?

What about animals, insects, viruses?

Is life unique, or an illusion too?

Or, are you the Chinese speaker, and the universe the room?

You seem to be arguing that because we can't do it yet, therefor it can't be done. Or that since we don't yet know how the brain works, that we can never know.

> Searle's answer is obviously not, from which he deduces that the computer program doesn't understand Chinese either.

Would Searle feel differently if he'd taken a computer architecture course?

There was something strangely magic about making an adder:

1. Learn about boolean logic, Karnaugh maps, etc

2. Define the inputs and outputs of a 4-bit adder, deduce some set of gates which would correspond to those outputs

3. Put those into a circuit similator and put a box around it

4. Suddenly the random AND and OR gates, which look arbitrary and none of which individually know how to do addition, collectively know how to addition.

Then you keep scaling that up: Put a bunch of gates together, none of which have memory, and suddenly you have flip-flops and registers. Add an instruction decoder and suddenly it starts zooming along doing things, executing simple programs that you feed it from simulated "memory". But you know that inside is just a spider's nest of logic gates.

I myself am religious and believe in a spiritual world distinct from the material world; but the Chinese Room thought experiment was never that compelling to me: I've seen spider webs of logic gates come alive as processors when assembled properly; I don't see an inherent reason why an algorithm of paper plus a very patient human couldn't come together to create something which "understood".

No, you're misunderstanding the Chinese room argument completely. It's not about scale, it's about the concept of "understanding" something. Here's another version that might make sense to programmers. I know Python, and I can read/write/compile Python code in my head. My computer's REPL can also read/compile Python code by following a detailed set of pre-programmed instructions to convert it to machine code. Nevertheless the computer does not "understand" Python, it cannot write Python code in response to any problem, nor does it "understand" Python the way a person does. It is not a programmer, and a programmer "understands" the language and can produce new creative meaningful output, and doesn't merely follow instructions.

It seems like the Chinese room argument is incompatible with a materialist world view. To a materialist, what could the brain be but a computer? Some sort of physical process is taking place inside that is is processing information, and at some level of abstraction that process will look like "dumb" symbolic manipulation. Yet we achieve "understanding"

Yes that is the point. The Chinese room is a thought experiment meant to prove dualism.

No it's not. Searle is explicitly not a dualist.

He thinks conciseness is a physical property. It's not clear to me why he thinks only brains, or maybe only living things, can have this property.

He makes the analogy that a weather simulation in a computer can't make anything wet so therefore a computer program can't have a thought. My take is that when we think about rain we don't get wet either. Human minds are the same sort of thing as the weather simulation, not the same sort of thing as the weather. For me thoughts are simulations, or models, or operations on models, and that those activities are tractable to computation.

How do we know that what we achieve is different from what the Chinese Room achieves?

Doesn't that beg the question?

Nothing about materialism implies lumping organic and inorganic substances into a catchall term "computer" leads to greater understanding of how either works.

The catch in your argument is that from the outside, you manipulating code and your computer manipulating code are indistinguishable. You say the only difference is that you understand, whereas the computer doesn't. However, from the outside, there is no difference between a statement print("I understand this") running on you or a computer.

What you are stating is the entire point of the thought experiment.

Cool, there are at least five different 'that is the whole point' takes on what the Chinese room means in this discussion. Also, I disagree with your take on it, that's not at all the point.

The fact that you’re saying “from the outside” is literally the point. Outside implies an inside implies dualism. Hence the thought experiment. If there was no outside the thought experiment inherently wouldn’t be.

Easy there with the words. Just because there's two of something is hardly an argument for a (mind-body) dualism [1]. Is the dichotomy cats vs dogs then proof for said dualism as well? How about vanilla vs chocolate? What do you make of the existence of a (six sided) die then?

[1] https://en.wikipedia.org/wiki/Mind%E2%80%93body_dualism

smh, have a good one.

> Outside implies an inside implies dualism

Well, unless with dualism you just mean that there are two, you are simply mistaken. Just because my car has an inside and an outside, and I cannot determine the state of its gearbox from the outside w/o the car telling me, does not mean that this implies a ghost in the machine. Far from it.

I think the Chinese room experiment is pretty effective, but a person might say the room is conscious and one might not. We all agree that, if we had a Chinese room in front of us, the only real way to tell would be a Turing style test.

Not really, Turing test and its variants aren't that relevant anymore. Even Turing himself didn't like it.

Why are they not?

It’s not a parkour trick when it is understood :)

Indeed, but is the human mind truly advanced enough to understand parkour?

Dualism has always seemed like a complete red herring to me. Even if you assume dualism is true, and that human consciousness has some non-material basis, what reason is there to believe that an AGI couldn't also have a conscious mind with non-material aspects?

I see this from people who expect that quantum computing will be an essential component of building AGI…

However a lot of these people are throwing everything at the wall to see what sticks when it comes to selling whatever quantum computing related tech or service they are working on. So it comes with a pile of skepticism from me. However the philosophical point of quantum computing being necessary if any of the “quantum dualist” type theories of consciousness turn out to be true.

I don’t think dualism necessarily means AGI is impossible for the reason you pointed out. But it would definitely mean it’s not inevitable, the non material part may be something that is somehow unique to people.

It depends a bit on what we mean by AGI.

Even if dualism is true, and humans have something special, that doesn't mean that a paperclip optimizer is impossible. A paperclip optimizer would be very, very smart (in the sense of being able to solve problems and achieving its objective of turning the universe into paperclips), but would not have a consciousness.

> Basically it comes down to whether we are purely material beings or whether there are non-material aspects to our mind.

The current state of physics suggests that we really have no idea what the "material" universe really is, at any scale. There may well be aspects of physical reality that we have yet to even conceive of. Given that when we cannot even say what a lump of rock is truly made of, or what even the dark void of space is made of, it should be obvious that we cannot be certain at all about what consciousness is made of.

I'm sorry, maybe it's just my tiny monkey brain, but I see no rational reason why someone would even consider dualism to be a thought worth entertaining. We obviously know material changes affect mental functioning directly. So how the heck could that substrate exist if not at the physical level?

Most of the confusion around the Chinese Room arises when people don't realize that "computer" and "computation" are not synonyms.

Turing defined computation as I/O behavior. The Church–Turing Thesis states that as far as their I/O behavior is concerned, all sufficiently general computational mechanisms (computers) are equivalent. The Chinese Room Argument presents a stupid computational mechanism with I/O behavior equivalent to a human communicating in Chinese. Some people then conclude that because the Chinese Room is clearly not intelligent and "because all computers are equivalent", computers cannot be intelligent. But that requires leaving out the qualifications from the Church–Turing Thesis.

But maybe there is more to computers than their external I/O behavior. Maybe intelligence is a property of the internal behavior of a system rather than its external behavior. Then there could be some computers that are intelligent and others that are not, even if they have the same external behavior.

For the love of god can people stop abusing the Chinese room?

Searle is a materialist, pretty hardcore one at that. What he wanted to show with the Chinese room is that simulation of intelligence and intelligence are not the same thing, that a machine, or even a human for that matter, can perform symbol manipulation without having any internal understanding of the thing they perform.

It's an argument against functionalism (i.e. that a digital computer 'thinks' the same way as a human merely because it performs the same tasks), not an argument for dualism. (that mind and matter are ontologically dinstinct substances)

But it doesn't. It totally fails to meaningfully define "understanding".

It defines it just fine. Understanding means grasping what a symbol refers to and what it represents conceptually, rather than just engaging in symbolic manipulation itself.

Just imagine a perfect rule book that already has a translation from one language into the other for every imaginable sentence, some native speakers just have written any possible combination down. You could 'translate' by just visually picking without having any idea of what's being said. They understand, you do not.

ok, so you define something that isn't understanding (symbolic manipulation, use of a magic rule book).

But where do you define what it is? "grasping what a symbol refers to" is another way of saying "understand" - it doesn't define what it means to "grasp".

There is some gray area in between where maybe AGI is dependent on some (material) mechanism that we haven't discovered yet, and hence can't be accomplished using present computational setups.

Absolutely, the "inevitable" timeframe may be millenia for all anyone knows, it just means that we will eventually achieve it given enough time and resources.

Reply on news.ycombinator.com