Comment by Isinlor | Hacker News Reader

Comment by Isinlor | Hacker News Reader

Parent Full thread

Small disclaimer: I don't want to take position on sentience or consciousness.

"LaMDA is not an agent with desires that it expresses in language."

Some counter points:

- being an agent is not a high bar to clear; a thermostat is an agent

- "with desires" - LaMDA can be thought of as rational agent whose objective function is to produce text with high likelihood

- "that it expresses in language" it expresses likelihood of next token in language

"it can't fear anything because there is no time for it to do so."

There is a time for it to experience something and that is when it does feed forward pass.

Also, as you correctly notices, LaMDA is not only doing greedy text generation, but also a deeper text completion search. But in fact just prompting a model like GPT-3 or Gopher with "let's think step by step" and letting it process in context is already significantly improving performance on wide variety of reasoning tasks [0]. I don't see how this is functionally different from human pondering.

The only objection that seems reasonable to me is that LaMDA is not correctly describing it's internal state, because it will happily generate descriptions of its internal state that we know are physically impossible. Like being a squirrel.

But I don't see how you can describe a system that will be fundamentally functionally more capable than closed-loop computation. LaMDA is almost certainly Turing complete. Postulating that it is fundamentally not capable of doing something is the same as rejecting Church-Turing thesis and postulating super-Turing computational model. That's a heavy claim.

[0] https://arxiv.org/abs/2205.11916

> LaMDA is almost certainly Turing complete.

I very much doubt that, and it is perhaps the key to our disagreement. If I believed LaMDA were Turing Complete, I would be more likely to think that there is even a small chance of it being sentient in some sense.

> - "with desires" - LaMDA can be thought of as rational agent whose objective function is to produce text with high likelihood

> - "that it expresses in language" it expresses likelihood of next token in language

I don't agree with both statements at the same time.

I can agree that we can say that LaMDA is a rational agent whose goal is to generate the most likely next token (or safe, full human-like reply if we look at the entire system).

But then, we can't say that it uses language to achieve this goal. An example of it using language to achieve this goal would be if, prompted with "What is your name?" it's output would be "Please help me answer this - what would a human think is a likely, safe answer to this question?". Instead, it will generate a sentence that it deems likely.

If we are modeling LaMDA as an agent whose perceptions are text prompts and whose possible outputs are text answers, than it giving a text answer that matches the prompt is more similar to an animal running away or howling in pain than to a human communicating. If the agent were sentient, we would expect to see higher-order behaviors, such as discussing the prompts instead of answering them, asking questions; or, at least generating answers as it is programmed but in a way where it tries to achieve more, similar to how an animal may act normally to get close to you, than snatch your sandwich from your hand (indicating that it had a plan and was displaying normal behaviors with a higher-plan behind them).

I'm highly confident that LaMDA can, or with small fine tuning should be able to apply Rule 110 [0]. People claim that LMs are just pattern matchers or hash tables. Rule 110 is literally 8 elements long hash table.

I'm not sure why you think that question whether LaMDA is using language to achieve its goals is relevant? Whether it's using language, tokens or floats seems to me just accidental.

> If we are modeling LaMDA as an agent whose perceptions are text prompts and whose possible outputs are text answers, than it giving a text answer that matches the prompt is more similar to an animal running away or howling in pain than to a human communicating.

I can agree to a comparison to an animal running away or howling in pain.

With regard to communication. I guess LaMDA doesn't have communicative intent besides providing likely completions. But communicative intent is not difficult to achieve. Act of communication can be modeled as cooperative hidden information game. Hanabi is that type of game, I believe there are computer agents that can play Hanabi with humans. They certainly do have communicative intent, many of the even have explicit theory of mind of higher levels.

> indicating that it had a plan and was displaying normal behaviors with a higher-plan behind them

Deception and planning is also achievable by current computer agents that play games like no-limits texas hold 'em poker on superhuman level.

LaMDA is probably no good in Poker. But LMs can kind of play games like Chess or Gomoku.

GPT-3 (and LaMDA probably too) also seems to be able to combine deception and theory of mind in a functional way:

https://twitter.com/JanelleCShane/status/1535835610396692480

I think it's exceedingly hard to formulate necessary condition for sentience.

I haven't seen a good formulation yet.

[0] https://en.wikipedia.org/wiki/Rule_110

> I'm highly confident that LaMDA can, or with small fine tuning should be able to apply Rule 110 [0]. People claim that LMs are just pattern matchers or hash tables. Rule 110 is literally 8 elements long hash table.

I would like to see that proved. I don't see why we should believe that LaMDA's training on a corpus of human text would help it guess that the correct output for a sequence like "apply the following rule to the input string 1101: [explanation of rule 110 here]" should be "0111". Even more so, I highly doubt it would be able to keep track of this enough to encode and execute even a relatively simplistic computation (say, computing the addition of 1 + 1).

I even more highly doubt that this would actually work with the entire system as Lemoine was given access to, including the facility of generating several possible outputs and comparing them for quality metrics to only output the best.

Still, even if this did work, see my next point for why it isn't what I was thinking of when you said you believed it is Turing complete.

> I'm not sure why you think that question whether LaMDA is using language to achieve its goals is relevant? Whether it's using language, tokens or floats seems to me just accidental.

All of the arguments I've heard for why we should believe LaMDA is sentient (while a CPU isn't) are related to the text it generates, "the way it answers questions about itself and its desires".

That LaMDA could be (ab)used to to generate some other kind of tokens that we could then interpret as a pre-programmed computation isn't that interesting - my CPU can do that to, and no one is claiming that it's sentient and that I should ask for its permission before asking it to run a program (in a more personal way than sudo :) ).

> I think it's exceedingly hard to formulate necessary condition for sentience.

I think it's exceptionally hard to formulate sufficient conditions for sentience, but I think communicative intent, theory of mind, and high-level planning are some pretty clear necessary conditions.

While it's possible in principle to combine various AI approaches to achieve this, I don't believe it has been done, and I doubt you could simply connect LaMDA to AlphaGo or some poker AI to get an AI that can explain its intentions in Go in words, or talk to other players to try to convince them it's not bluffing.

Reply on news.ycombinator.com