Hacker News Reader

Comment by lukev | Hacker News Reader

As someone who has worked/is working on building LLM driven agents, this allegory basically reads to me like:

1. I ask a LLM for paperclips. 2. ~Magic happens~. 3. Humanity is doomed.

Even if a LLM was good enough to drive a system like this (current ones are not, by a long ways), there's a million reasons it would not and could not play out this way.

View on HN

I think (from the outside looking in) many lesswrongers are slowly coming around from the "paperclipper" scenario to something more gradual, in response to how the tech seems to be developing currently. More along the lines of "we hand over bigger and bigger parts of running the world to AI systems and this gradually leads to less and less human involvement (until such time as the intelligent systems that run the world see humans as an unnecessary interference that can be done away with)". Something like that at least.

Some pointers here, maybe: https://astralcodexten.substack.com/p/davidson-on-takeoff-sp...

I mean, to me this sounds like "some christians are coming around to the fact that cosmology is mostly correct about the shape of the universe, but they want us to say that God created the big bang since you can't explain it yet."

Which is to say, right or wrong, it's a narrative in search of confirmation more than it's an actually rational understanding of what is real.

IMHO you're doing it a disservice by calling it a "narrative" and comparing to christian cosmology. On the face of it I think ideas like "past some threshold AI can become dangerous to humans" or "AI can become more intelligent through a positive feedback loop", even if speculative, have some plausibility and are worth investigating.

You are basically chiding people for changing their mind in the face of new information, while still staying committed to some core ideas based on argumentation which is not challenged by this new information. Sure, this roughly looks like what happened to religions as science developed, but it looks just as much like how any scientific or philosophical field of inquiry makes progress.

Given that the "lesswrongers" are just a bunch of pontificating blowhards, why should we take them seriously on anything?

The basic idea of the essay, I think, is that "an agent" is a thing in idea-space that a non-agent intelligence (oracle) has access to and can describe to a user. "describe" in this context means "write code to represent" Then the user is turning what was an oracular AI into an agentic AI. The idea here, is that if you think that you're safe because you haven't made an agent, just an oracle, that's a false sense of security. Oracles are just one user choice away from becoming an agent. If one thinks agents are safe for some reason, that's a whole separate discussion I think.

Reply on news.ycombinator.com