Hacker News Reader

Comment by drKarl | Hacker News Reader

Article makes sense, but what if by pausing any further research on AI then Russia or China get ahead and actually end up building AGI?

View on HN

How does it make sense to assume that this is all leading towards a hostile entity?

It's not hostile in the sense of hating us, it's just a maximizer, and when you're smart enough and try to maximize one thing (or some function of multiple things), anything that's not in your utility function gets sacrificed. Humanity maximized for cheap energy, and for a long time, the carbon composition of the atmosphere was not in our utility function. Now that's clear we might be killing ourselves by doing it, we're working it in, but if it turned out that CO2 was only dangerous to, say, several species that we didn't care about, it's very likely we'd just accept it as the cost of progress.

Why not just train an AI to care about humanity then? One problem is in defining actually what it means to care about or preserve humanity. Another problem is creating an AI that actually internalizes this as a goal, instead of just saying the things we want to hear until it gains enough power that we are not a factor in its plans anymore.

It's probably possible to align an AI, it's just not a problem we've solved yet. We've seen many, many, ways in which things can go wrong with less-intelligent AIs, and there's no reason to think that things will just magically somehow be easier as the AI gets more intelligent.

The artice is literally about how this will lead to a hostile superhuman intelligence that will wipe out humanity. It's not about AI tooling as an accelerator, which is a valid concern that imo can't be avoided.

Humans aren't that hostile, but we've driven many, many species to extinction or endangerment in trying to get the stuff we want. We destroyed habitats for farmland, hunted whales for oil, tigers for fun, elephants for ivory for our pianos, etc. Why do we like music, why do we want pianos? We just do. Evolution is a process that only optimized for reproductive fitness, to make us want to reproduce and pass on our genes, but we learned that we like music just because. We also don't all care that much about passing on our genes -- people stop having lots of kids once they get richer, some people don't even want kids at all. The way we train these AIs is a bit different from evolution, but it shares the characteristic that we don't have a lot of control over what the model learns (<https://en.wikipedia.org/wiki/AI_alignment>).

An AI we train to take care of us will almost certainly learn to want some other stuff, in addition to not exactly caring about us the way we want. That's fine if it's not too smart, we just stop it from doing dangerous stuff. The problem is when you crank it up to a million, and then those things that are just a tiny bit wrong become a massive problem, and you can't turn off the AI because it is smart enough to understand that the things it wants to do won't get done if it's off, and it doesn't want to let itself be modified in the same way that you wouldn't want someone to give you a pill that makes you stop caring about your children.

I think the idea is that it doesn't matter who builds the AGI first if we all end up dying anyway.

And the first step to prevent that from happening is for us to stop, and then hope that the other powerful countries will see that and stop as well. And everyone who's left can be forced to stop by the powerful countries who agreed to the treaty.

Yeah, I get that, but I what if Russia and China think "Well, we're fucked anyway, our population is aging faster than any other country in history, our economies are fucked by multiple factors including COVID, War, sanctions and technology bans, our countries won't make it past 100 more yeah, maybe not even 50 more years... if we achieve AGI before anyone else, or no one else does, there's maybe a 5% chance we can control it, and maybe a 60% or 70% chance it destroys all humans on earth or all life on earth... if we're lucky and we hit that 5% we'll rule the rest of the world, otherwise we were doomed as a country anyway..."

Then they end up racing to build an AGI we're not prepared for, and we all die.

The faint hope is that people are not crazy or suicidal enough to build a world-ending technology, that humanity will manage to see ourselves as being in the same boat, where we all live or all die as one. And just-not-do the thing that causes our extinction.

It's not a very realistic proposition, because even smart people often fail to understand these arguments (or just disagree with them), let alone the politicians (who are incentivized to ignore them), or the rest of the people. There's a lot at stake for the "winner" of the AGI race, who could build an AGI and align it successfully (which they can't do). And many countries are ruled by people who are clearly willing to do crazy and suicidal things for power.

That's why the author of this post doesn't put a lot of hope into the idea of his daughter growing up.

Yep, it's always a possibility, but it's better to reduce the chance any way we can. If the US banning it reduces the risk from 100% to 50%, that's fantastic.

I also don't think it's that likely that a leader would take those odds, if they understand the risk fully. The worry is that they won't understand or believe the risk, so that's where we should attack the problem, in my opinion--by putting out as much good information on AI risk, and combating misinformation, as best we can.

That's addressed in the article.

The way it's addressed is wishful thinking

The only solution is to treat large GPU clusters the same as nuclear enrichment facilities by rogue nations. They have to be destroyed.

Reply on news.ycombinator.com