Hacker News Reader

Comment by labrador | Hacker News Reader

Bard: The author argues that ASI is likely to be developed soon and that it could have devastating consequences such as

    1 The development of new weapons that could easily destroy humanity.
    2 The accidental release of harmful substances or organisms.
    3 The displacement of humans from the workforce, leading to widespread poverty and unrest.
    4 The loss of control of AGI, which could lead to them making harmful decisions

1, 2, and 3 were concerns before AI, but we've managed so far. 4 seems to be the one that is most terrifying, but then we shouldn't be putting AI in control of nuclear weapon launches or other potentially harmful activities

View on HN

I'd only say we've managed 1, 2, and 3 so far because these issues typically are being controlled by humans in human timeframes and scale so we have a grasp of the situation and stop it before it gets out of control.

Where number 2 becomes iffy is the 'accidentally on purpose' reengineering of the biosphere via burning carbon sources for energy. Overfishing and extinction of numerous species isn't exactly what I call good in the managed so far list.

AI is already in control of any number of harmful activities, and will become more entrenched as time goes on. If you're watching the Ukraine war and the rise of drones as massively impactful in theater operations you'll see where the future is going. Massive numbers of inexpensive but semi-smart and potentially connected devices overwhelming the enemies defenses are what's in every military planners mind right now. Then you have the groups that are trying to figure out how to counter such types of attacks, most likely with their own sets of smart drones. Do you think we are going to have humans controlling hundreds, thousands, tens of thousands of these devices at once? Seems unlikely to me, it will be passed off to GlaDOS or some other controller system where people put in the basic goals and the AI figures out the rest.

Even AI screwing up something like global shipping can have deep and impactful economic outcomes, imagine an actual attack on it. Humanity has never been more fragile than it is now.

None of these four points are part of the author's focus. Broadly, I would summarize the author's points as:

- Alignment and capabilities research are not separate. There is an "alignment dilemma" on whether to contribute to alignment work or abstain from it.

- Discussions of AI risk are subject to "persuasion paradoxes" that make it difficult to reach clarity.

- It is helpful to take a "recipes" framing: (a) what are recipes for destroying the world? (b) how likely is an AI to discover or enact these recipes?

Overall, a clear post.

It's clear, although I think the "recipes for ruin" focuses too narrowly on the lower bound for ways to destroy the world, like a bright teenager with $10,000. Another important framing is "what degree of economic resources might an AI be entrusted to manage, in the future, and are there any recipes for ruin that fall within that range". For example, if an ASI is put in charge of managing an investment fund with $10 billion in assets, would it be capable of building a doomsday device (and disguising it as a productive investment)?

Agreed, but devastating consequences are my interest

Enriching uranium is hard. "Synthesize this gene sequence" is easy and potentially more destructive.

If you believe the lab leak theory for Covid, I assume that variety wasn't created with AI. Humans will continue to be the most dangerous creatures on Earth for a long time, not AI alone. Humans with AGI could be even more dangerous, so I can understand the push to regulate AGI. I'm afraid it's a losing battle. "When AGI is outlawed, only outlaws will have AGI"

Reply on news.ycombinator.com