Hacker News Reader

Comment by lolinder | Hacker News Reader

Many people instinctively assume that researchers in AI are the best qualified to comment on whether the current line of research is likely to pose existential risks. If you assume that then the situation sure seems dire, with many of the most prominent researchers warning of the oncoming apocalypse.

However, there's another possible interpretation: AI researchers chose to pursue their field because they believed true artificial intelligence was possible and wanted to pursue it. They steeped themselves in the literature of superintelligence and set that as the target, which primes them to buy into their own field's hype.

This is their life's work: of course they believe in it. We should be careful not to take their expertise in deep learning's foundations as evidence that they have expertise in where it will end—no one can know that yet, and they're the people who chose this path of research because they believed in it before anyone else did.

View on HN

They do have particular insight into the trajectory of capabilities; less insight into how the newfound capabilities will interact with the world; and no particular insight into values.

The usual failure mode with experts is that they overstate how important their field is and demand money to fund further progress in their work. It's at least slightly interesting that this particular group of experts aren't self-aggrandizing in an attempt to make more money and get to develop their research interests further. Instead, they're arguing that their research needs to be frozen and severely regulated.

In my opinion, the existential risk of AI is very real. The issue is that policymakers lack any real understanding of the issue, and most regulations will focus on mostly benign things (what if end users do something naughty with it?) and not the central existential risk (ew that's nerdy and alarmist). You can see that dynamic with Newsom.

We'll end up with the worst of all worlds, where regulations prevent valuable small-scale use cases that improve our lives, but do nothing at all to prevent existential risk. Maybe even making the existential risk worse, though that's more speculative.

Many people arguing that AI risk is real have big monetary incentives. Some are asking for money to study safety and influence the regulatory bodies. Others gain money because believing in AI superintelligence makes their AI startup look like a great investment. The true believers like Bengio are a smaller subset.

> Others gain money because believing in AI superintelligence makes their AI startup look like a great investment.

And still others are hoping that the fear will lead policy makers to build up a regulatory moat that they'll be able to navigate but their up-and-coming competitors won't.

It remains highly interesting, and indicative of something, that researchers in other fields orient almost entirely on the "give us money to improve our technology," not "give us money to research how to fix our technology, and pause it in the meantime."

Reply on news.ycombinator.com