Comment by AlexErrant | Hacker News Reader

Comment by AlexErrant | Hacker News Reader

Parent Full thread

Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.

He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.

Why do you treat unpredictability as neutral? If something has a potentially existential, but completely unpredictable outcome, surely one should avoid heading for it?

Like, knowing that there is a limited number of plausible doom scenarios and what these are, would make me more confident in taking the risk than settling on "it's literally the singularity, we can't know what happens beyond".

Perhaps I'm not being clear.

Doom-post-RSI is still very much possible and should be avoided. It is not neutral. If I had a high-variance coin that would either kill or uplift humanity, I would not flip it. The status quo is preferable to extinction.

I am being _descriptive_ in my comments. I am not being _prescriptive_. Nowhere am I telling anyone what to do or what to think: quite the opposite. I am simply saying things that, to me, are obviously true.

I'm perhaps being... loose with my language when it comes to knowing things post-singularity. E.g. we can still discuss scenarios in which the takeoff is fast/slow, or what would be necessary to bring robotics/electricity/logistics to the wider world, what the limits of intelligence are (you can't RSI your way to a bioweapon if you lack a wetlab). That kinda thing. And it's even likely that RSI will be spikey: why would RSI in, say, machine learning, necessarily lead to superintelligence in, say, writing fiction? Or cooking a Michelin-star meal?

My main point is that doomers saying SUPER INTELLIGENCE IS ENOUGH TO KILL US ALL is baseless. That's all.

Reply on news.ycombinator.com