I mean, I'm worried about climate change, but not because it's fun to worry about. And I put a lot of my time and energy into thinking about how to do something about it effectively, but not so much because it's fun. Sometimes I'll hit on a fun idea in the process, but I'd be much happier if I didn't have to worry about it at all, but that's not really a choice I have.
I'm not sure how much I should worry about AI, but I'm certain that for the people who are worried about it, it's because they're convinced it's a threat, and they aren't having fun.
I've met a lot of these people and regardless of how you feel about the sincerity of their convictions, they are definitely having fun.
> On the morning of September 12, 1933, on a miserable, wet, quintessentially English autumn day, at the intersection where Russell Square meets Southampton Row, Leó Szilárd waited irritably at a traffic light waiting for it to change from red to green. He had just attended a lecture by the great English physicist Ernest Rutherford. Rutherford, known to many as the father of nuclear physics, was discussing the newly prophesied release of energy from atoms, most notably by science-fiction pioneer H G Wells in his book The World Set Free. In his baritone voice, Rutherford, acknowledged master of the atomic domain, dismissed this fanciful idea as nonsense. Any thought of releasing the energy locked in atoms, he said, was "moonshine".
...
> What Szilard realised as he stepped off that curb was that if we found an element that when bombarded by one neutron would release two neutrons, it could lead to a chain reaction that could possibly release vast amounts of energy
https://blogs.scientificamerican.com/the-curious-wavefunctio...
Secondly, you can have a diverse portfolio of causes you care about, and Effective Altruism has recommendations for areas including global health and animal welfare. Taking AI risk seriously doesn't mean you can't give time and money to other causes with more tangible results.
But if you've spent the past decade arguing precisely the opposite - that instead of spreading philanthropic dollars over broad portfolios because true efficacy is unmeasurable, they should be directed to causes which provide hard evidence that they're among the most efficient use of a marginal dollar, and that charities that aren't conducting RCTs to prove how much better off people are after being fed aren't really evidence-based enough to recommend - it's going to raise eyebrows when you pivot to recommending people also give their money to opaque organisations with vague goals and zero evidence they can achieve them, whose only thing in common with the highly evidence-focused aid agencies is that their pitch is nerdier than the average nonprofit. I mean, there's nothing particularly wrong with "I give to causes because their pitch speaks to me" as a heuristic for philanthropy - it's how most altruists think - but it's not how EA pitched itself as being different from that and caring about efficacy and evidence.
The Wells analogy is awkward for the AI safety crowd for another reason too. Wells' central thesis that an atomic super weapon would guarantee a peaceful, utopian world turned out to be horribly wrong. We got actually used bombs, arms races, proxy wars and very nearly used bombs instead, and that's without Nazi atomic scientists getting a heads up. To the extent his writings on the subject of atomic weapon safety had any effect at all on atomic weapons, they made them more imminent and more likely to be used. If marginal dollars spent on promoting the distribution of The World Set Free would have had any effect at all on the probability of armageddon, they would probably have increased it.
On your first point, I can donate towards anti-malarial interventions and towards reducing AI risk: two very diverse causes, but both under the EA umbrella. Diversifying amongst recommended EA causes seems to me a good bet, especially given I expect some of them to require pivoting away from as more research is done.
Perhaps we can summarise the last decade of EA into something like: "given the available information (largely based on RCTs, research and an understanding of moral philosophy), we expect that money spent here will do more good than money spent there. More information will change this expectation."
We now have information that there is a potentially very grave risk confronting us from AI. For other cause areas we can use evidence like randomised controlled trials to inform our actions. In the case of AI risk, this type of evidence is unavailable, but there are nonetheless other types of information available to us that suggest it is worth taking seriously. This information takes the form of compelling arguments from credible people in the AI industry and the industry's achievements to date. Of course we'd like to have firmer footing, but the nature of this risk doesn't permit it. We can either ignore it, and potentially walk to our doom, or we can take it seriously and try to reduce the uncertainty and the risks.
I agree with you that this raises eyebrows, but I don't think it's a gross departure from the last decade.
I think it is wrong to say we have "zero evidence" that organisations committed to reducing AI risk can achieve their goals. I think it is more accurate to say it is highly uncertain that they will be able to achieve their goals. My view is that we should try to address AI risk because the stakes are high.
I would have sided with Albert Einstein in trying to renounce nuclear weapons, even though we have "zero evidence" for humanity ever coming together to do such a thing in the past.
Of course - nuclear weapons really do exist and AI is only a hypothetical threat at the moment. My point is that there was "zero evidence" that the world could come together to voluntarily control a technology, and yet people like Einstein tried to make it happen, and with good reason. Dismissing this is like taking Rutherford's view in the 1930s and saying there's very little point worrying about nuclear weapons because they're not possible, and it's even more pointless thinking about how we might control them.
Finally - why might the pitch speak to nerds in particular? I think in part because it is based on Bayesian reasoning, which is both nerdy and incredibly effective in situations of uncertainty.
There's nothing unique to EA in this (people convinced that most aid money is wasted sometimes give to organisations they have some sort of ethnic/political/religious affiliation or people they generally like unconditionally, and sometimes that trust heuristic has even been right!) but it did purport to be the school of altruism that didn't.
As for the Wells analogy, the two most notable uses of AI safety funds seem to be (i) attempting to build AIs and (ii) distributing fiction they believe best illustrates the possibilities of AGI, so I'm not sure I'm altogether off the mark in imagining what well-intentioned utilitarian nerds of the time (Wells certainly was one) might have done with more cash. If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal for a class of technologies which evolved self awareness from being rewarded for meeting human needs, then popularising the idea of conflict between human and AI could be more likely to have extremely adverse unintended consequences than intended ones. Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
Can you let me know more about that?
> If you subscribe to the view that I do: that "secretly unite to eliminate humanity" is a fairly non-obvious and risky goal
I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
> Popularising the idea that it's the self aware general intelligence that's the dangerous bit about AI, not the intermediate step involving humans having access to immense calculating power whose recommendations they don't really understand seems like something of an existential risk blindspot too.
It sounds like you think AI is risky too - even if only at this intermediate level? I'm curious - do you have a view on how we could go about reducing this risk?
Compare the level of transparency and need for funds of OpenAI LP with the average organisation criticised by GiveWell for not being transparent about how they spend the money, not providing any metrics or trials, and/or having more money than they need to fulfil their short term goals and plenty of access to more.
> I don't think AI would have the goal "secretly unite to eliminate humanity", but I do think that most of the goals we can imagine it having can play out in very bad ways that achieve that outcome as a side-effect. There is an incentive for it always be looking for ways to maximise its power (compute, resources, money, influence), and this could happen in ways we really don't want.
Is "people might not recognise how the second order effects of pursuing a goal adversely impact humanity" a problem unique to decisions made by AI? If anything, I would have thought that in addition to actually existing, independent human intelligences' resource demands were already more competitive with human needs than a silicon-based AGI's is particularly likely to be, and demonstrably neglectful of second order effects. With those problems mostly unsolved, it's hard to see tackling the same thorny problem but with hypothetical goals of hypothetical entities would be seen as the more impactful line of research (as interesting as it undoubtedly is). Nothing wrong with intellectually curious altruism, but if you're describing yourselves as the arbiters of effectiveness...
> I'm curious - do you have a view on how we could go about reducing this risk?
I don't have an easy solution to a range of complex mostly political problems, no, and certainly wouldn't argue for investing in studying them on the grounds of other philanthropy being less effective. Generally attempts to impose bans and developing more, better weapons quicker seem to have been attempted solutions to non-AI based arms races and neither of them have a great track record. Misplaced faith in the output of current techniques we call AI as superior reasoning free from human bias rather than garbage in garbage out mathematical transformation skewed by massive unseen flaws in training sets and prone to non-obvious bugs isn't likely to be reduced by talking about allying with self-updating superintelligences...
Isn't it substantially worse than this? The EA people generally are, or are closely tied to a lot of the AI risk people, right?
Similarly, a large fraction of people at the Nucleic Acid Observatory (where I now work) are EAs, for similar reasons: becoming convinced that this is what most needed doing.
So many words just to say "they don't value the exact same stuff I value".
You're free to put your money where your heart is, just like them. It's not right to hit back at those who are doing what's right by their own account.
This argument is appearing again and again in this subthread. "Putting your money where your heart is" is the exact opposite of what EA claims to believe. EA is about investing effort and money into areas with the highest marginal value to humanity.
So if they're spending their time talking about the robot invasion, we can only conclude that's where they think that effort should be spent.
Obviously they're free to do whatever they want with their own time and money. But if they spend that time pretending that SkyNet or WOPR is the greatest threat to humanity that the rest of us should be spending our marginal effort on, it's on the rest of us to call them out on that nosense.
And it's the individual who determines that? Or does your own personal opinion override everyone else's world views?
[1] "Responsibitility", i.e. not just my "right" in a liberty sense! Again, the theory behind EA is that we should collectively decide on the most effective altruistic actions based on rationalist principles. Those principles absolutely include "you're wrong about the robot overlords thing".