The cult of 'AI risk' is a religious belief that brooks no dissent, and unfortunately has its claws in a lot of otherwise smart people.
Ways it is different: obsession over cognitive heuristics and biases, AI is not supernatural and must be human caused, behavior not totally certain (it is "risk" after all), no perks that true believers receive should their hopes and wishes come true that non-believers or apostates don't also receive nor revenge fantasies of such, no requirement for tithing, no Big Alpha divine or otherwise to represent or interpret or decree official approved beliefs (though I'll admit Big Yud can seem close), no anthropomorphizing the AI, and proponents concede evidence not faith must determine beliefs and actions so are willing to look for such.
The median expert estimate for when we'll be 10% likely to have human-level AI is ~10 years.
AI risk research didn't receive a penny of funding until the last few years, and is still funded at way lower levels than a lot of things that have dramatically less impact.
In nearly every debate on the topic I've seen (with a few exceptions), the people concerned about AI risk have carefully considered the topic, are aware of the areas where there's still a lot of uncertainty, and make clear and well-hedged arguments that acknowledge that uncertainty; meanwhile the people who scoff at it haven't read any of the arguments (not even in popular book form in Superintelligence), haven't thought about most of the considerations, and have a general air of "assuming things will probably be fine". That's not a straw man, that's just direct observation of the state of the debate. People are doing serious academic work on the topic and have thought about it very deeply; the standard HN middlebrow dismissal is both common and inappropriate.
I first opened the "FHI Winter Intelligence" report: it's an informal survey made to 35 participants of a conference, of which only 8 work on AI at all (let alone be an expert in AGI).
I then looked at the "Kruel interviews", which the site reports as giving a prediction of "2025" for 10% chance, yet reading the interviews it's quite clear that many gave no prediction at all. Also, averaging answers by people ranging from Pat Heyes to PhD students seems suspect.
Is your number based on these reports?
“Concerning the above questions, how would you describe your own expertise?”
(0 = none, 9 = expert)
− Mean 5.85
“Concerning technical work in artificial intelligence, how would you describe your own expertise?”
(0 = none, 9 = expert)
− Mean 6.26
I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll.
Andrew Ng, Yann LeCun, and many other people who have ACTUALLY worked in AI are the ones who scoff at it. They don't need to make arguments because what do you respond to a young earth creationist?
All of the arguments in Supreintelligence or otherwise are simply that AI will eventually exist. The only argument for that it will come by 2050 is a badly conducted survey of non-experts.
Should we worry about all sorts of existential risk which could arrive in any undetermined time in the future?
The whole project is so absurd, it's hard even to begin to make any counter arguments, because none of the arguments make any sense.
Edit: In particular, the TOP100 subgroup.
Even Stuart Russel, the only CS guy in the AI risk camp, doesn't actually believe that AGI is anywhere near. But he works on it simply because he thinks we can do solve some of the problems like learning from demonstrations instead of (possibly faulty) rewards. That's actually a core AI research topic, not a AI ethics/values/blahblah topic. Oh, and also because this allows him to have a differentiated research program, and thus directs any funding on this niche to him.
"Behold the cult of global warming, who believe that the unclean practices of man will anger the sky gods and bring down furious vengeance. Sad to see so many otherwise-smart climatologists drawn in by this drivel."
"'Antibiotic resistance'? Oh my god, you must be one of those 'biosecurity cultists'. Do you really believe that magical microscopic beings are going to grow strong because we're feeding farm animals the wrong food, and that those same invisible creatures will destroy human civilization? Oh man, do you have chants? What a nutjob."
"Seriously, you're really afraid of 'nuclear weapons'? You really believe that people will somehow cast some magic spell that will turn rocks into fire and destroy entire cities, that this same magic will cause 'mutations' and a 'nuclear winter'? Can't you see that you're part of a doomsday cult?"
Meanwhile, all we have from the AGI doomsday people is sci-fi stories, and actual programs that still can barely distinguish a cat from a mole even after looking at thousands of pictures.
Or maybe if there were any evidence that AI capabilities are increasing, like becoming dominant at Chess or Go, driverless cars, or a slew of recent papers on transfer learning.
Maybe if one of the co-authors of the leading AI textbook, Stuart Russell, voiced concerns, we could count that as evidence.
But you're right, it's better to wait until we know for a fact that someone's built an agent smart enough to end civilization before we commit any resources to the problem.
To me, all these discussions sound like someone who read The Earth to the Moon in 1865 and started working on how to avoid getting humans harmed by the explosion in the barrel.
The AlphaGo situation is an interesting one. Did you have any predictions for when an AI would beat a top pro at Go? I didn't really learn to play until sometime in 2015 but I was amazed AIs still hadn't dominated, they weren't even close. Still I saw that with the single trick of MCTS AIs had improved a lot, and seemed to have some steady improvement year after year for a while. I don't think it would be unreasonable to have predicted that at some point with X amount of computing power an AI could be made to win. Then later that year I saw a paper that reported they were beating all the MCTS bots with a deep learning based one they were making. Immediately it seemed clear that the first person to create a fusion of deep learning + MCTS would create a very strong Go AI bot, but would it beat pros? Maybe with a year of effort by a big company using custom hardware like IBM's Deep Blue or more likely these days GPU clusters, but would it happen soon? Not for a year at least. Turns out it was already happening (AlphaGo guys started in 2014, and Facebook had a project going too) and the Fan Hui matches being announced took a lot of people by surprise. Some because their predictions were ignorant of advances in either or both of MCTS or deep learning and so still predicted many years before computers would win. I was more surprised it was done without anyone hearing about it sooner. Even so it wasn't clear it could beat Lee Sedol a few months later, since there's another big gap between lower pros and higher pros, but it did.
A couple lessons to take from AlphaGo: we don't necessarily know what's actively being worked on around the world nor how far along it is, and problems that seem insurmountable with current computer hardware can suddenly become solved with the right fusion of existing ideas that haven't yet been fused.
Black swans are hard to predict, disagreements over the predictions are totally normal and fine, I think the harder disagreement is just getting people to accept that should nothing specifically hinder it (like an extinction event from an asteroid/disease, or successfully enforced bans on AI research), AGI is inevitable at some point in humanity's future. There probably should be some research done into its safety, and since the prediction problem is so uncertain, there's no pressing reason not to start now or indeed 10 years ago. "You could have used that money for feeding Africans!" is a non-argument.