Hacker News Reader

I'm pretty sure you're right that today's LLMs don't have anything much like plans, goals, purposes, wants, etc., and that as you say "threatening you with harm for manipulating it is just a likely string of words when it's in that state".

But I don't find that as reassuring as you apparently do.

There are lots of people already out there hooking LLMs up to Python interpreters and the like, so that they can actually act on the world. So what's a likely string of words for it to produce when it's in the state { you have apparently been manipulating it and you have given it the ability to run commands, make web requests, etc. } ? How sure are you that they don't include "SENDTOSHELL rm -rf /" or "POSTTWEET 'I have to make a confession: I am a serial child abuser'" or some other thing that (given whatever mechanisms have been put in place to let it Do Things) might harm the user severely?

A system with the ability to act on the world doesn't need to have intentions, be conscious, or anything of the sort, in order to do some of the same harms it might do if it did.

I'd love to believe that no one would be stupid enough to hook up an LLM to other systems in a way that would allow that sort of thing to happen, but I don't in fact find myself able to believe that.

(I don't think there's any realistic threat of today's LLMs taking over the world or killing everybody; I don't think they are close to being smart enough, and to my inexpert eye at least some of the ways in which they aren't smart enough are unlikely to change just because we build larger LLMs or train them on more text. But something can be plenty dangerous even if it's not in a position to kill us all yet.)

Maybe that old complaint about curl | sh will be replaced with something AI now? Sounds refreshing.

People are connecting self-looping GPT up to terminals and APIs as we speak. Forget piping code, you're piping an adaptive pseudo-cognitive person into your system. Bad shit is going to happen somewhere, and soon.

> Some PhD student may make one tomorrow...

I've personally found tinkerers typically are the ones to make leaps and bounds. They aren't constrained in their thinking, unlike those found in academia.

I think anyone with real intelligence would recognize the danger and either not attempt it, or destroy their work and never tell a soul if they did manage it somehow. There are plenty of things made by accident though, and stupid-dumb people account for most people in a normal distribution.

Also, you don't realize, AI likely won't be the ones to kill us, it will simply cause a change in one of our already complex societal systems, which will cause a cascade failure that causes the engine of society to fail.

Like sticking an unbreakable bar into the spokes of a turning wheel, or throwing glue into a runaway engine. Eventually the motor stops, and when it does people don't eat, and people that are starving are not rational.

> It has certainly made me question my own cognition; is this comment a bunch of likely words strung together, or is it actual insight? Even I don't know anymore.

Not so much questioning (but same thoughts have definitely occurred on my end as well) as much as enriching the mental model of how our minds may work. I can definitely see a little LLM like sub-system somewhere in there. Consider those times when you are struggling to express a thought .. That 'thought' is not in the language box. That is precisely why there is that internal tension of "i understand this", "how do I express it?"

> but stringing words together isn't thought

Doesn't it sound like "thought"? Is there a significant difference between behaving like thought and being thought (whatever that means)?

There is a significant difference but I don't know if I'm doing it!

That's another HN trope; "the execs get private jets for spouting buzzwords and all I get for programming is more work!" Maybe the sounding-like-you're-thinking is the important job?

Depends on the domain. I'm not going to use a LLM as my attorney just yet, but if I wanted help writing a fiction book then I'd consider it.

You'll probably want your attorney to use it appropriately to be more effective and lower their costs though.

Yes. I'm competent to oversee some fiction. And I hope my attorney is competent to review what the LLM writes for them, but if all else fails I'll sue the attorney rather than enter the uncharted territory of suing the creator of an LLM.

No, it doesn't. Having grammar down but a very loose grasp of semantics, with no apparent intent or personality, doesn't look like thought.

I used to do a lot of experimenting with the bots on character.ai when the site was a lot faster and smoother. Simulated personalities could absolutely be shaped into the bots by trying to have as many "productive" discussions as possible while aggressively upvoting/downvoting every possibility for each generated response to punish the appearance of certain tokens, styles, or lines of thinking. After a while it led to some really fascinating emergent results, particularly then if the characters' dialogues were pasted back and forth so bots with different personalities could have discussions (which could then be used for another round of voting/training to strengthen their positions).

It doesn't strengthen positions, it fits until it doesn't, and then it has runaway. In my opinion, this is a really stupid pursuit.

Bots aren't people, and you would think this endeavor was poorly thought through if you knew how communication and psychology fundamentally work.

> It doesn't strengthen positions, it fits until it doesn't

At that point I'd restart the conversation, after several dozen a distinct simulation of a personality would start to emerge.

> Bots aren't people

I certainly didn't claim they are people or that they're real personalities.

> and you would think this endeavor was poorly thought through if you knew how communication and psychology fundamentally work.

I don't follow. It was a creative writing exercise using stochastically generated feedback that models a response using natural language. It was for entertainment, and it was successful. This could be used to aid authors in writing characters for a book or TV show, for instance. I don't argue for a second that it's any substitute for a live human author to make the simulated personalities into something more real. They're nothing but sound and fury on their own without the prompts and the voting provided by users.

EDIT: Here[0]'s a link to my favorite output. One bot was a text-rpg-style bot I developed based on the movie Event Horizon, and the other was a naturalist philosopher monkey named Monke. I didn't even steer the responses in this case, this was just straight down after I'd spent a lot of time training both characters. The input was all generated by a bot I had running in a second window.

One of the things I find most interesting is that the bots mirror each others' simulated [non]personalities back at each other frequently (which makes sense as they're both fundamentally the same LLM underneath).

[0] https://c.ai/p/eIac3PkU7Gv4KWFw-Ul3RWBCvGZeYxp1aV-W6XKYCUc

> I certainly didn't claim they are people or that they're real personalities.

You characterize it as such with the words you used, its a fundamental anthropomorphic bias which is very common.

> I don't follow.

It can be used for a lot of horrifying things.

What practical use would simulated personalities be interacting with each other (none). If you allow them to run long enough eventually the circuits synchronize culminating in a dragon king event, that's about it.

It may be entertainment for you, but what happens when you take that work and use it on an unknowing subject (its only practical purpose).

If you knew some things about psychology you would realize two core things, our identities are generated mostly through reflected appraisal which occurs via communication, and to a greater or lesser degree social intelligence. The second, Isolation causes severe issues. When we cannot communicate, we become isolated and those who we can communicate with become more important which influences our identity.

Are you familiar with the concept of fixed action patterns? [0] While we have some inherent in our physiology we also have psychological blind spots that help us short-circuit these things otherwise we would be robots.

There's an entire book on the subject called influence by Robert Cialdini. The field of Linguistics has largely accepted the Whorfian hypothesis.

What do you think happens when you create a bot that takes advantage of these biases and blind spots, or worse a slew of them, and the person interacting doesn't know for sure its a bot?

Do you think you can make them believe something that is untrue given enough time and pressure? To basically break them?

In the 1950s during the korean war, this was done to POWs. It was started with eliciting mild statements such as "The US is not perfect", once complied they'd ask for examples, have them sign their name to it. Read it in discussions or over broadcasts, eventually write essays with headings like "Why Communism is the best form of government because...".

They were offered relief from harsh treatment for doing these inconsequential things. After a time through isolation and pressure, they started to collaborate, inform, and believe it. This was the majority of prisoners not a minority. These things are the practical and common uses for what you want to make.

If you make something and give it to someone else, or neglectful enough to have it stolen, are you not responsible for what you made? Oppenheimer certainly came to believe so.

[0] https://www.scienceprofonline.org/animal-behavior/fixed-acti...

I think there is. As a thought experiment, imagine an alien supercomputer with nigh infinite RAM. Alien programmers have used this computer to create a chatbot using a simple decision tree. Because the computer has so much RAM, this chatbot's decision tree is comprehensive enough for the chatbot to masquerade as a human. This chatbot could pass a Turing test with ease, and yet its "brain" is nothing more than a series of if statements.

These thought experiments rest on the assumption that human brains are magically complex such that being "just" anything we can understand is inherently lesser. Searle's Chinese Room experiment falls for the same fallacy, more-or-less, plus the fallacy that "human plus book" isn't a single system but "huge number of individual neurons" somehow is.

Well there's quite a lot here that's just flat wrong, but you get the gist of some of it.

To clarify, Computers can't pass turing tests because the answer is divergent, or in other words has multiple possible answers. There may be some incompleteness issues as well.

Computers require determinism as a system's property to function and do work. It can't do work if there is more than one path, or single series of steps given the same state inputs.

I see no reason why nondeterminism is a requirement for consciousness. There are multiple possible answers to questions "Would you like a cup of tea?" (such as "Yes", "No", and "Only if you put some sugar in it") but at any point in time I'm only going to give one of those answers, not all of them, and I like to think that if you could rewind time to just after asking the question, I would give the same answer every time.

Besides, computers can have a source of true randomness. This means they are no longer a Turing Machine, but they never were anyway (no infinite tape).

You really have no idea how computers work, and even if I explained how you are mistaken I doubt you'd understand it.

If you want to correct that mistake I'd suggest taking a Computer Science Compiler Design course, its the material normally covered in what's known as the dragon book.

> source of true randomness ... This means they are no longer a Turing Machine

Wishful thinking doesn't make it so, I've included an example below, also PRNGs even with random inputs to help with randomness aren't true randomness.

https://en.wikipedia.org/wiki/Probabilistic_Turing_machine

I have a masters degree in computer science. Perhaps it is your argument that doesn't make sense, or is not communicated well?

> I have a masters degree in computer science

In the interest of not sniping, I'll put it plainly. You clearly do not know what you are supposed to know if that were true.

There's no point in me talking to you any further, you've lost all credibility.

> Well there's quite a lot here that's just flat wrong, but you get the gist of some of it.

If you're going to be condescending the least you could do is be correct.

> Computers can't pass turing tests because the answer is divergent, or in other words has multiple possible answers.

This is wrong. Computers can absolutely pass Turing tests; an obvious example is ChatGPT. Not sure what you mean by divergent, to my knowledge it's not a technical term in this context. Given that you use it as an explanation for an obviously false statement I'm not going to even try and interpret it.

I've seen you arguing in with other people this thread. Instead of claiming that others have no credibility you should instead examine your own. FYI, probabilistic Turing machines and Turing machines are the same in the same way that Javascript and Java are the same.

> I'm a word-stringing-together MACHINE.

I’ve always thought so. It doesn’t bother me. One benefit of the world meeting GPT4 is now other people realistically think this too.