Hacker News Reader

Comment by bordercases | Hacker News Reader

An argument that makes all of its relevant assumptions clear and using precise distinctions would move "fairly obvious" and "seems to be so sure" and "sounds naive" into crisp statements about our knowledge. That's why it's desirable. Its use goes beyond simply you being convinced.

View on HN

In the case of arguments about the dangers of AGI, there is no meaningful distinction between assumptions and conclusions. The points being discussed are so fundamental that they don't lend themselves to being broken down further.

If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere.

The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise.

Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.

> The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition.

In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.

That's because you know what Stockfish's goal is. That's not true for a hypothetical AGI.

That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds.

It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2].

[1] https://intelligence.org/files/Corrigibility.pdf

[2] https://www.deepmind.com/publications/safely-interruptible-a...

Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q

Are you sure you can't be more specific? Wasn't MIRI studying the math of alignment precisely because one should be more specific about alignment? In your view, were they successful or misguided or what?

My view is that the entire "friendly AI" movement, and indeed the very idea that it is even possible to align an AGI, is sheer hubris.

Assuming the qualities of AGI are what its proponents claim (namely, intelligence far beyond the upper limit of human intelligence), humans "aligning" such an entity is laughable. We've had spacecraft built by highly intelligent engineers lost because they overlooked a metric/imperial conversion, and similar people believe that they can devise a watertight scheme to effectively contain an adversarial superintelligence?

The smartest humans can barely contain very small aspects of the regular Universe, and the Universe isn't even targeting them like an AGI would.

Reply on news.ycombinator.com