Comment by dotsam | Hacker News Reader

Comment by dotsam | Hacker News Reader

Parent Full thread

> Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.

If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.

It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).

The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.

I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.

If a hacker designs an AI which has among its talents "self-replicating to another host", did an AI escape containment, or did a hacker write a virus?

The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.

I definitely don't want to sound like I'm saying we should stop worrying about hackers and bad actors, they are of course a concern. I certainly don't want to advocate for dismissing real dangers and power structures in the real world in favour of speculation over some theoretical risk. But I do want people to think about it and take it seriously, in the same way I would have wanted people in 1933 to take seriously the possibility of nuclear weapons even though they were then only theoretical.

I think the sticking point in this is that you don't think that AGI is ever possible - is that right?

If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".

I'd then feel much better about spending some time researching the topic in more depth.

Reply on news.ycombinator.com