Comment by chubot | Hacker News Reader

Comment by chubot | Hacker News Reader

Parent Full thread

Yeah this is basically what I was thinking after reading the article, thanks for spelling it out concretely.

My thought was that if you're going to assign infinite weight to unlikely possibilities, then why not devote your entire life to preventing time travel from being invented in the future?

Someone could invent a time machine and destroy the Earth before humanity had a chance to exist, etc.

It's just about as falsifiable as what they are working on.

---

Also, if you're going to be so abstract about it, then why you wouldn't want to optimize for the happiness and future vigor of AIs? If you're going to be a vegan, then presumably you care about non- human consciousness.

So what's wrong with a future with flourishing AI and without humanity?

There are too many ridiculous things to count here, and normally I would ignore it, but it seems like people are taking it seriously and it has consequence? (And the article is pretty well written either way)

---

It's weird that he had doubt that's like 1 sentence in the article, and then entire lifetimes of work that simply ignore the doubt:

My No. 1 worry is: what if we’re focussed on entirely the wrong things?” he said. “What if we’re just wrong? What if A.I. is just a distraction?

And as alluded to in the article, the wild swing from helping the poor to preventing AI has a huge moral hazard. With the former you can course-correct your actions based on observation of outcomes -- what happened as a result of the donations?

With the latter you can justify any action today by just pushing out the hypothetical benefits further and further into the future.

It resembles a religion that collects money today, promising benefits in the afterlife.

> If you're going to be a vegan, then presumably you care about non-human consciousness. So what's wrong with a future with flourishing AI and without humanity?

Not necessarily anything, but a pretty mainstream view now is that dangerous AIs need not be conscious (and indeed, that AIs can still be dangerous even if it's somehow impossible for an AI to be conscious). So you could imagine futures in which an AI conquers the world but has no experiences and can't either flourish or fail to flourish.

A computer virus is a familiar existing example of software that can be dangerous but that could probably not be said to flourish, even if it spreads widely, accomplishes everything it's programmed to do, and persists indefinitely. So we could envision dangerous AIs that are like today's computer viruses but that are also programmed to cause some kind of effect in the outside world, and that have some ability to learn and adapt so that it's difficult to prevent the effect and difficult to remove or deactivate the virus. That could be pretty dangerous to have active anywhere on the Internet, but still probably couldn't be said to flourish.

> Someone could invent a time machine and destroy the Earth before humanity had a chance to exist, etc.

I struggle with assigning infinite weight and unlikely possibilities. But we can stay in the every day world and ask what credence do we place on the time machine possibility, and why? All information I have read suggests that it is essentially impossible, and so effectively I don't worry about this contingency at all. And then, is there anything we could do to prevent the time machine scenario? I can't think of anything at all.

On the other hand, to focus on one particular risk, there are compelling arguments made by experts in the field of artificial intelligence (e.g. Stuart Russell) that say that we face real risks based on the current trajectory of the technology. Is there anything we can do to reduce the risks? That seems like an important question.

It is important to be sceptical, and whilst I don't place 100% credence on the likelihood of a truly catastrophic AI outcome, I do think it is worth taking this possibility very seriously. There is a strong chain of reasoning behind the claims, and they are worth engaging with, especially given the fact that we are in a position to influence the outcome.

> if you're going to be so abstract about it... what's wrong with a future with flourishing AI and without humanity?

We can park the abstract side for a moment and think concretely about such a future. I want humanity to flourish, particularly during my lifetime. Credible people are currently ringing alarm bells about the potential destructive power of a technology we are actively developing, and so I think on balance it is better (at a minimum) to listen and be vigilant than to dismiss it. Better yet, I think we should engage with the arguments and see if there is anything we can do to reduce the risk of a potential bad outcome.

> On the other hand, to focus on one particular risk, there are compelling arguments made by experts in the field of artificial intelligence (e.g. Stuart Russell) that say that we face real risks based on the current trajectory of the technology.

To the extent that these concerns are not overblown, one way to avoid the dangers of advancing ML would be to stop all the effective altruists [1] in the field haring along at building it irresponsibly.

The people preaching about the potential future dangers of AGI are the same ones pouring money into ML; if they actually meant what they said, they'd pour that money into their opponents.

The current practical dangers of ML are concerned with enforcing social inequalities and concentrating power in the hands of unethical actors. The fact that many of the unethical actors claim utopian reasons for their reckless disregard of consequences is not a reason to believe them.

[1] https://twitter.com/emilymbender/status/1556691543850831872

Reply on news.ycombinator.com