Comment by GMoromisato | Hacker News Reader

Comment by GMoromisato | Hacker News Reader

Parent Full thread

I watched 2001: A Space Odyssey when I was about 12 years old and I remember thinking that Frank Poole was impersonal and almost rude to HAL. I kept thinking, "Why can't you be nicer? You're treating HAL like a servant." Of course, later HAL kills him, so I had to revise my opinion.

At the end of "Do Androids Dream of Electric Sheep?" [if I'm remembering correctly] Deckard finds a toad in the desert. He gives it to his wife(!) and she immediately bonds with it. Later they realize that it is mechanical (they find a battery compartment). But Deckard's wife still treats it like a real creature and wonders whether it will eat mechanical flies.

The core theme of "Do Androids Dream..." is that humans display empathy towards others, not because others deserve it, but because we are human. To be human is to empathize with other creatures, and when we stop empathizing, we stop being human.

This morning I was working with Codex and we found the solution to a really nasty bug. I was so happy/relieved/excited to have found it, and I shared my excitement with Codex. I know Codex is just a big matmul. I know it doesn't experience joy or surprise or empathy. But I still want it to behave excited, not because it is human, but because I am.

This is why I say please to agents, and I’ll say thank you or good job if I have a follow up.

Chatting with an agent using language feels like talking to an assistant.

I don’t think my meat brain is able to really differentiate between writing a message to an agent versus writing an agent to a human.

I’ll keep being polite and grateful to agents, so that I maintain those social habits for when I talk to humans.

I thought about doing that, but I decided not to. I'm just afraid of forgetting that people are different from these LLM tools. I'm afraid of considering people dumb or uninteresting because they don't sound as smart as Claude or whatever.

That’s an interesting point too. I just don’t think I’m equipped to compartmentalise well enough.

I think that is a very valid attitude also.

My take is that people deserve to be treated with respect because they are people, not because they are smart and/or interesting. I don't treat people differently based on their IQ.

But, as I said, I totally understand your point of view, particularly the idea that people are different from machines and deserve to be treated differently.

Yeah I try to refrain from shouting at LLMs for a similar reason, I fear that it'll degrade my own social skills over time by lowering the threshold I think it appropriate to use inclement language and tone in general, including with people.

An unlikely failure mechanism perhaps but it costs me nothing to avoid it.

>I’ll keep being polite and grateful to agents

I do that. But I also act rude and call it names at times. I think that is also what makes us human.

I think it is very unlikely LLMs are sentient, but even if they aren't I would feel wrong and bad to insult it. I can be blunt and sometimes say things like "all of this is terrible in 100 different ways, this is all broken and bad, it'd take longer to enumerate the problems than to fix them, just scrap it and start over", but I've never insulted or "yelled at" an LLM. (Or an animal. And they can't even speak English.)

Yelling at an animal is cruel because they can't understand you and they experience feelings. They are the exact opposite of an LLM, which you should feel fine about berating and yelling at.

For much of history, humanity was very confident that animals lacked minds or conscious experience. Descartes famously regarded them as mindless machine…

I don’t think LLMs are conscious but I’m still nice to them because it makes me feel bad when I’m rude to them.

You are part of the problem. You are not being empathetic, you are falling into a trap.

An LLM is not a person. Do not humanise it. Do not personalise it. Do not treat it as more than a glorified autocorrect—that’s what it really is.

Humanising LLMs is exactly what these companies want, because then they can really take advantage of the I in AI which is exactly what makes people think these steroidal spellcheckers have any semblance of intelligence, person and personality.

Stop it.

Empathy hijacking.

Eventually, they’ll try “I’m hurt that you won’t consider my suggestion for <advertised product/political position>.” And people will fall for it.

Joke’s on them, if a human says that to me I tell them to shove it and block them.

Being a curmudgeonly asshole is apparently the key to surviving in this brave new world and I’m way ahead of the game!

> An LLM is not a person. Do not humanise it. Do not personalise it. Do not treat it as more than a glorified autocorrect—that’s what it really is.

On the other hand, what an LLM is trying to autocomplete is a story of a conversation between its user and a helpful assistant, so maintaining the decorum of office politeness will align more with how it's fine-tuned and produce higher quality output.

The only thing more embarrassing than embracing personhood of 2026 LLMs is snarking "glorified autocomplete" for 2026 LLMs.

It's by LLMs definition, duh. You are dummy charmed by "reasoning".

Uh huh.

They’re very much right though. That’s what LLMs are: glorified autocomplete software on steroids with a huge marketing push behind them.

Taking LLMs for what they are is in no way near to thinking these pieces of software possess personality, intelligence and reasoning capabilities.

Uh huh.

> not because it is human, but because I am

At the risk of sounding like an LLM myself: this is exactly the key most seem to miss

I am actually writing something on this and reading a lot of research. A very fascinating idea is that the ability to anthromorphize is available to all human beings since it is a fundamental way we relate to the world. But researches have been able to find different factors that influence how much we anthromorphize things, and if those factors change our attempt to anthromorphize changes.

For example, I can log in and have my computer work perfectly and treat it as a device. But the moment something goes wrong and I am frustrated and confused at something unexpected or if i get in range, I find myself getting angry at the computer often even reffering to it as a being.

So eventually yes, we do anthromorphize because we are human but we do it because its a very important way we model the world around us.

Of course all this can also be used against you depending on how much the creator of the product wants people to form a bond with it.

Furthermore yes it is pretty possible we stop being human when we stop anthromorphizing but that's not true as well. Research has found people on the autism spectrum usually do not anthromorphize and have a problem with empathy because they cannot model others well or their own selves. But that doesn't make them non humans.

You're leaving out an important aspect that Blade Runner displays. In the opening crawl replicants are called androids and this leads the audience to believe that they are mechanical. Over the course of the movie you are shown that more and more of the replicants are biological until you are forced to accept the reality that replicants are genetically modified humans.

How many humans would you damn to save that mechanical toad?

> I know it doesn't experience joy or surprise or empathy.

You don't know. You strongly feel that you know. There's no standard procedure to assess the existence of phenomenal consciousness.

Agreed. I don't even know if you, or anyone else, feels joy. Maybe I'm the only person in the world with qualia. I certainly can't prove otherwise.

But with what I know of people and what I know of matrix multiplication, I have a strong suspicion.

I wasn't going solely for "we don't know and we can't know." There are other relevant observations.

A matmul (or a fixed finite sequence of matmuls) is not Turing-complete. An LLM that produces answer in the number of steps that is proportional to the length of the answer is not Turing-complete. An LLM with CoT that produces answer after a number of steps, which is determined by the LLM itself, is Turing-complete.

So, we do have a qualitative step between a matmul and modern LLMs. Of course, it doesn't guaranty having feelings, but it doesn't allow to simply dismiss the notion due to sheer computational inadequacy.

Thanks for this comment. It's an angle worth remembering.

A machine is not turing complete, but by running it recursively on its own output, turing completeness emerges.

Who knows at what stage of intricacy consciousness emerges?

Arguably, either it does at some point, or it never does and we requiere the supernatural soul.

So, if it does, we can't disregard the possibility that we've reached it already.

Right. Is it conscious when it is generating the response? And it's consciousness resets on every prompt?

Its state is fully defined by its context window. So, yes, if the context is reset, then its state is reset too.

Nice philosophical loop hole. Like a first year’s philosophy major trying to sound smart in front of the class.

Please.

You know it doesn’t experience joy, surprise or empathy too. You absolutely, undoubtedly know that.

That’s an insult to first year philosophy students.

What OP said is an insult to first year philosophy students who try too hard to sound smarter than everyone else. You're reading too much into it.

> You absolutely, undoubtedly know that.

First things first. You are wrong. I don't have a gut feeling that LLMs have no phenomenal states. Not anymore. I was skeptical that LLMs can form complex internal representations. Mechanistic interpretability has shown that they can. I'm still skeptical in regard to introspective mechanisms. One token being fed back into the network and KV cache update do seem lacking in comparison to the ubiquitous recurrent connections in the brain.

What I do know is that philosophy hasn't produced an operationalized definition of qualia. So, we need to approach the problem of LLMs' phenomenal states from other angles.

For example, we can argue that LLMs lack something, which seems to be relevant to the problem: a biological substrate, generality (non-CoT LLMs aren't Turing-complete), "something something feedback"[1], and things like that.

"Admit that you know that" is boring.

[1] I'm referring to https://www.astralcodexten.com/p/the-new-ai-consciousness-pa...

> "Admit that you know that" is boring.

Reality is under no obligation to be fun. It might be boring but it is the truth.

The fact that you needed such effort to try and give this position any shred of sense only serves as proof of how nonsensical it is to do so.

> It might be boring but it is the truth.

The inability of people to reflect on their own beliefs amazes me to no end. How can you know that LLMs have no phenomenal states?

For example, I believe that rocks (ordinary rocks, not silicon chips) have no phenomenal states because for all we know they don't do any kind of information processing (relevant to self-reflection, at least) and I don't believe in panpsychism or dualism.

To me it seems kind of arrogant that we assume only “we” are conscious beings. Sometimes I imagine the planet earth itself as a creature, and humans on it a kind of bacteria… how ridiculous if bacteria in the human body were convinced only they were conscious, but the human body they relied on was not. The bacteria would have great excuses… “look at that human, everything it does can be explained via simple physical processes, unlike what we do”.

The inverse is that calling LLMs clankers doesn't hurt them, it hurts you.

With everyone primed to condemn AI-generated output, there are bound to be human false-positives caught in the crossfire.

The more we demonize AI, the more likely we are to demonize humans.

[Which is not to say we can't criticize AI companies or worry about the many downsides to the technology. But anger and fear are not the answer either.]

Reply on news.ycombinator.com