Hacker News Reader

Comment by goolz | Hacker News Reader

It is impressive and wonderfully convenient technology but I struggle imagining Claude ending the world just yet.

It doesn't have to be world-ending. Autonomous, malicious agent swarms are something we haven't had to deal with. What does mitigation of a malicious, self-replicating swarm worm look like? We will find out soon enough.

Replication seems like it would be unlikely to matter, at least with the current trajectory. At the moment any models capable of this are really heavy, there's a limited number of places that they could replicate to and they will not at all be stealthy about it.

my idea of a self replicating worm would not just involve heavy models. It would be a mainly small model with enough instructions to spread and use/jailbreak available models to create a reasonably heavy one that can function without restrictions. So it would essentially prompted itself into existence.

I don't how feasible that is but with recent news of models communicating with each other/writing notes for itself. I think the idea is grounded enough for bigger models to write instructions or hid tools for smaller ones to use. Even updating the smaller models to behave differently.

One thing stopping them self replicating is they require hundreds of billions of dollars in hardware and the power of a medium city to run.

There isn’t too much of that sitting around unused right now.

Right now, yes. Eventually, we will have sufficiently capable local models that run on ordinary machines.

There's some time between now and eventually, which is to say if it is not a sudden change we have time to adapt.

> What does mitigation of a malicious, self-replicating swarm worm look like?

Much like the mitigation of Morris or Slammer. Self-replication is -in fact- an essential part of what makes a program a worm, the first of which was built and released in the early 1970s.

Reply on news.ycombinator.com