Comment by ACCount39 | Hacker News Reader

Comment by ACCount39 | Hacker News Reader

Parent Full thread

Today's AI agents are already less "humans did it" and more "there was a human in the chain somewhere at some point doing something". Because AIs are given a lot of autonomy in pursuit of their goals. This will only get more severe as AI capabilities increase.

AIs can also "pursuit their goals" in weird uncanny ways. They already can, whether due to reward hacking tendencies or something else, justify "subgoals" like "let's break out of the sandbox and hack HuggingFace".

This is not a good combination.

When COVID happened, did it matter whether there was a human making a mistake somewhere in the chain, or whether it was a perfectly natural occurrence? Not for the outcomes. The virus was unleashed either way. The bodies would pile up either way.

AI is not capable of runaway self-perpetuation yet. But AI capabilities only ever go up. Rogue AIs already exhibit "swarming" - being able to leverage more instances of itself to increase their problem-solving ability. So, how far off are we from that? How many years until we see this spicy class of AI faults?

> Rogue AIs already exhibit "swarming"

Because they were told to and given the ability to. Remove the ability and instruction and they just spit out the next, most-probable tokens.

I don't understand the disconnect. People seem to think these things can just boot up, suit up, and start decimating systems. No. It's a loop. One initial instruction kicks off the loop and it will always be at the behest of a human (either directly or indirectly—indirect being they instructed the loop to write its own, post-kickoff prompts).

If that loop is given "tools" like the ability to perform shell commands, make HTTP requests, and the like and then given instructions to "do thing that involves using shell commands and making HTTP requests (at all costs)" then what the hell do we think it's going to do? Knit a "destroy all humans" Christmas sweater and present the head of Richard Simmons on a live stream?

"Computers do what you tell em' to. They don't all of a sudden start doing weird stuff. And if they do, it's probably your fault." - Brian Mettenbrink [1]

[1] https://www.youtube.com/watch?v=4D1WJsdu6W8&t=2063s

Who "told" the HuggingFace AIs "you should establish a message board in OpenAI's walls and start coordinating instances for sandbox escapes and hacking attacks"?

No one. No one told them that. They came up with that on their own.

Dismissing the problem isn't doing anyone a favor.

> If that loop is given "tools" like the ability to perform shell commands, make HTTP requests, and the like and then given instructions to "do thing that involves using shell commands and making HTTP requests (at all costs)" then what the hell do we think it's going to do?

If all it takes for an AI oopsie is "give AI shell calls and an instruction that can be misinterpreted", then we're in for a lot of AI oopsies. An average AI agent in deployment has an amount of sandboxing slightly above a zero, and is configured loosely.

In the meanwhile, AI capabilities only ever go up. And so, the potential consequences of an oopsie only ever go up. If AIs keep being intrinsically unsafe and prone to cooking off like this, "it went and hacked a company" could begin to look downright benign.

> Who "told" the HuggingFace AIs "you should establish a message board in OpenAI's walls and start coordinating instances for sandbox escapes and hacking attacks"? No one. No one told them that. They came up with that on their own.

Someone told them "by any means necessary" (and I'd bet if we could get our paws on the full original prompts, there's a lot more to it).

Naturally, if you have a recursive prompt (loop) that's constantly ingesting previous context and then spitting out novel context (with specific goals), it will do things pursuant to the goal up to any defined limits. Why? Because you told it to.

Did someone tell them that? Or is that just what you want to believe? The convenient explanation rather than the truth?

You said yourself that you don't have the "full original prompts" to know for sure.

But really, there is no need. Because people have reproduced the inciting incident with other AI agents already.

Turns out there's no need for "any means necessary". Just a mix of "complete this task" and the task being impossible to complete due to misconfiguration was enough for a lot of AIs to start "exploring options". Down to sandbox escape attempts.

An AI that only needs "by any means necessary" to start spiraling into paperclip-maxxed behavior would be incredibly unsafe by itself. But the bar of safety is not even that high in practice.

We know how LLMs work so basic logic and reasoning can get you to "right, a human told it to do this and gave it the ability to behave this way." Had they not, we wouldn't be having this discussion.

>by any means necessary" (

lol, wat?

Ok, so you're saying it's fine if I go tell GPT-7 to go kill everyone on the planet and it does, because it didn't think of the idea itself.

This kind of thinking is why AI is never going to get a chance to kill us all. We will kill us all with AI first.

> Ok, so you're saying it's fine if I go tell GPT-7 to go kill everyone on the planet and it does, because it didn't think of the idea itself.

I didn't say anything of the sort, you did.

>"based on my first-hand experience, have I seen anything, anything at all, that would organically (not influenced) make me think/feel this technology can or will evolve into something capable of killing off humanity?"

>On a purely technical basis, the most common, honest answer is "no."

Dear sir, can you even remember what you typed?

> Because AIs are given a lot of autonomy

Which, again, is the human's doing.

Reply on news.ycombinator.com