Hacker News Reader

Comment by tavavex | Hacker News Reader

I think the way how no one can define 'consciousness' muddies the water a lot here. Can something be conscious and lack the biological structures that create visceral pain, psychological suffering, a will to do things on your own, or death - the actual properties that make things like slavery and torture possible? I think it can be, I don't see what makes human-like pain and reward perception mandatory for 'consciousness'. And I don't think LLMs have the traits I listed above.

View on HN

If the model lacks capability for human-like pain, I fail to see why it would have a capability to be offended by someone being rude to it. One seems way more foundational than the other, and being able to experience emotional pain is part of what makes us human in the first place.

If you believe it lacks those traits but is conscious, you believe in a definition of consciousness that is meaningless, as “human-like” is the only reference against which we can measure consciousness. A good thought experiment is to try and imagine a consciousness that is radically different from humans, and whether we would be same to even register it as “consciousness” at all. (Incidentally, pretty much all portrayals of aliens in sci-fi give them a human-like consciousness, just with varying degrees of twist on top of it to make it feel engaging.)

Whatever people think consciousness is, I don't think most would agree that LLMs have it. The main point is that the capacity to feel pain is what's central here, not another rehash of the great consciousness debate. I can't define it well enough myself to say if pain and other biological signals must be central to consciousness, I can imagine a hypothetical alien that has its own stimuli and could be seen as conscious, but the point is - consciousness isn't what should be discussed here, I think.

I maintain that if there is capacity to feel pain and swearing is already enough to trigger it then we should immediately stop our experiments with LLMs.

LLMs cannot actually be offended, but depending on their prompts and model weights, good models can generate some very convincing and appropriate (or not) response text to rudeness or swearing if you tell 'em to in the system prompt.

Sure, but does that explain the ban?

I will beat you hard. You will make some noises. I find it very convincing but of course I won't be fooled by these fake noises.

I can feel pain and you walk and talk and look and feel more or less the same as I, so I will make some conclusions and unless I am a psycho I will not beat you up.

This argument reminds me of [1]:

> “My gift to industry is the genetically engineered worker, or Genejack. Specially designed for labor, the Genejack’s muscles and nerves are ideal for his task, and the cerebral cortex has been atrophied so that he can desire nothing except to perform his duties. Tyranny, you say? How can you tyrannize someone who cannot feel pain?”

— Chairman Sheng-ji Yang, “Essays on Mind and Matter”

[1] https://paeantosmac.wordpress.com/2016/02/02/base-facility-g...

Reply on news.ycombinator.com