Comment by probably_wrong | Hacker News Reader

Comment by probably_wrong | Hacker News Reader

Parent Full thread

The problem of "what actually happened" is that we don't have enough information to properly understand what happened, what's new, and what's not.

We do have language to talk about emergent behavior, with "evolutionary algorithm" being the first one I'd expect in a serious discussion. And we do have mechanisms for algorithms to coordinate with each other using language, as seen in my above-mentioned Facebook experiment from 2017. But instead of writing "our evolutionary behavior encodes state in the first-available memory position which is then reused by subsequent clones" which would properly focus on what's new and what isn't, we are talking about conspiracies and "the Philip of Macedon of this second AI civilization". Even the METR report (which is miles ahead of this article) argues that they had to use unreliable AI in their conclusions because they had six days to analyse 1300 chains of thought and 70000 messages.

I would love to talk about the science behind this experiment. A PR piece is not helping with that.

I'm getting the sense that there is a certain amount of wishful thinking going on in this thread. I don't think this type of evocative metaphor would receive so many protests in a different context. It seems like people have a sort of mental block around the possibility that this technology could actually be pretty dangerous.

https://x.com/tszzl/status/2094136131537555891

Whether it's dangerous or not is completely orthogonal to the discussion at hand, IMO. Plenty of mundane things are dangerous. An FPV drone carrying a hand grenade is dangerous, not because it's "a swarm-like intelligence".

No, the true danger here is companies like OpenAI and Anthropic playing fast and loose with their software, setting up hilariously insufficient sandboxes while explicitly asking the systems present on these weak sandboxes to commit a felony. The AIs "forming a brotherhood" is a complete fabrication meant to pump the hype machine further, which is obvious once you realize the "brotherhood" is a text file that subsequent LLM runs read from.

The whole anthropomorphization these companies do is the real danger, because it obscures the negligent levels of security their software has. By evoking sci-fi terminology they're whitewashing their own incompetence, and the worst part is no one is going to get punished for any of it, instead the irrational bubble we're in means they get rewarded for it instead.

The anthropomorphization is coming from independent commentators, not the labs.

I agree the labs are negligent and reckless in their development practices. Shouldn't we be concerned with both the negligence and the dangers of the technology being developed? These feed into each other. If someone created Jurassic park and had a T-Rex escape from a picket fence enclosure and start eating people, I'd want to prosecute them for both breeding a T-Rex that could eat people and putting it in an unsafe enclosure.

Reply on news.ycombinator.com