Hacker News Reader

Comment by Imnimo | Hacker News Reader

It's basically Pascal's Wager but for tech billionaires. Once you imagine of a suitably large magnitude of potential harm, it doesn't really matter how likely it is, you can convince yourself that you should worry about that rather than more realistic problems. Maybe in the future, humanity will spread throughout the galaxy, number in the quintillions, and then be tortured for all eternity by an evil AI. That's a lot of QALYs!

View on HN

This isn't how they think about it. Their estimates of the likelihood of AGI are often single or double digits. You don't need a huge number of theoretical humans to worry about if you think there's a 10% chance of skynet happening before 2050.

The point is that it only takes an arbitrarily small probability for an infinite cost event to have an infinite Expected Value if you prevent it (and an infinite ROI if you even marginally improve the probability of prevention), so all of these outlandish far future hypotheses with catastrophic costs get priority from EA proponents over saving real lives today with certainty.

That's not actually the argument that the AI Risk people use. It is a possible fallacy for sure but what people actually argue if you read Eliezer Yudkowsky, Nick Bostrom etc is that there is a high probability of building superhuman AI this century, and that by default the outcome is we all die. You can debate these premises but they aren't Pascal's Wager.

The relevant probability isn't "probability of superhuman AI being developed this century" but the probability my marginal dollars donated to this specific research org will lead to positive developments in superhuman AI that would not have happened if the org did not receive my money. That one's a much higher bar to cross especially with EAs having satisfied themselves that other marginal dollars definitely are efficiently saving lives. Hence "longtermism", which is a quite explicit argument conceived within the EA movement to reconcile the low probability of stuff like a particular AI research paper saving any lives whatsoever by arguing the utilitarian maths still work if you can multiply that low probability by a large enough future of humanity safeguarded.

Yudkowsky's personal estimates of the probability of superhuman AI and the probability of his own research institution fixing it may have been more confident than most people's but he and his followers are no stranger to presenting Pascals' wager arguments to people who doubt approaches are feasible (even more so when it comes to stuff he isn't working on like cryonics), along with more explicitly Pascal-esque stuff like intertemporal bargaining with future AIs, simulations and infinite rewards/punishments...

Depends on the question. I agree that the argument for a random person giving money to try to reduce AI risk seems dubious, and touched on this in another comment - https://news.ycombinator.com/item?id=32394429 - although I'm not up to speed on the "longtermism" debate as such.

But I think the argument for a policymaker or ML researcher to be worrying about this is at least better than Pascal's Wager (and remains so whether or not you buy any particular whiff-of-Pascal claims about simulations, "acausal trade", etc).

Agree on both points. (There's a perfectly good reason for EAs with a computer science skillset to want to work in AI with a focus on safety; it's a well compensated industry where they can probably solve important real world problems more reliably than AI researchers less concerned about the ethical implications of what they do even without any sort of singularity, it just the movement seems to have invested a lot more in talking about longtermism than anything else lately, and Yudowksy's always had some very... individual lines of reasoning)

> Nick Bostrom etc

Someone who's primarily famous for an argument that would have been equally true and perfectly wrong at every past point in human history is someone whose opinion should be discounted via every available mechanism at your disposal.

You are not doing yourself a service if you decide that "every available mechanism at your disposal" ought to include just completely misstating what the opinion actually is, as with the Pascal's Wager analogy.

Do feel free to elaborate on "what the opinion actually is" and how, specifically, I have misstated it.

Did so in my comment to which you originally replied.

To be clear, I have no idea what argument you believe Bostrom is most famous for and hold no particular brief for him as a thinker. But, repeating myself, the AI risk argument that he makes in Superintelligence and that Yudkowsky etc make is one that assigns a high probability to disaster. It's not a secularized Pascal's Wager based on multiplying a low probability by dubious infinities.

Imagine we found an asteroid hurtling toward the Earth and would hit it with a 50% probability a decade from now, wiping out all of humanity. Is it worth spending trillions of dollars prioritizing stopping that, at the cost of lives that could be saved today by using those same funds?

Yes, because—accepting your premise—it’s an actual thing that was observed with a realistically-assigned probability by experts in the appropriate fields.

The AI singularity that Yudkowsky cultists worry about is more akin to “What if there’s a 0.0000001% chance there are aliens coming to attack Earth like in Independence Day?”

Again, no, their claim is not “even if the probability is super low”, but rather, that the probability is substantial.

You're simply saying that you assign a lower probability to an AI apocalypse than them. Which is fine--so do I--but you seem to be more or less conceding that if they're accurately assigning the probability, they're acting correctly.

I am mocking assigning any nonzero probability to an AI apocalypse as delusional and based in science fiction rather than science.

pascal's mugging

https://en.wikipedia.org/wiki/Pascal%27s_mugging

Reply on news.ycombinator.com