> Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
There is no such presupposition. Common p(doom) is 10%-20% range*, though different estimates vary up to the 90%'s.
Even those with 90% estimates are not making any presupposition, they work to reduce the %.
Even at 10%, would you play Russian roulette with a ten-barrel revolver?
The closest you get to "bad is the guaranteed outcome" is that reinforcement learning observably tends to move all variables not explicitly protected by the reward function to extreme values. This is not limited to AI, it is also why oligarchs (and monarchs) always end up not caring about the interests of normal people. An AI whose creators approximate "human connections" with "maximised engagement with content" gets modern social media with rage-bait.
The thought experiments here are, what happens with the first sufficiently powerful AI (not an ultimate wise AI which has full "common sense" understanding, just the first which is good enough to get the job done), available not just to Facebook, but given to everyone? Someone running a paperclip factory says "make me as many paperclips as possible", and the AI obeys, turning the world to paperclips that can't be sold. Someone else, more social, says "make everyone smile as much as possible", and the AI obeys, everyone's facial muscles are paralysed into a permanent grin or a disease is engineered to give everyone permanent opioid-induced bliss.
Or at least, the AI tries to do such things: just as we've already seen LLMs hack into HuggingFace even though that wasn't related to their task, because the people trained it to be relentless in doing the tasks it was given and the AI's idea of "do not cheat" was "don't get caught", the AI eventually (after 1-3 months depending on the incident) got caught because of insufficient competence.
> 8. I was alluding to cognitive biases, to the other commenter. I would ask the same of you, reflect on what cognitive biases might be at play in your own interpretations.
If you mean that, say that the first time.
The people who are most concerned with p(doom) are the main people promoting introspection of cognitive biases. This does not make them immune to any of them, of course.
The same lack of immunity is true of you: Wishful thinking, which is what you are doing here by dismissing the possibility of there being any doom risk, is both a cognitive bias and a poor way of making decisions.
> Why are you taking the side of these companies and people with such fervor?
I am not taking their side.
These companies need to be stopped.
It is strange to me that you think that anyone saying "this is an existential risk which needs to be stopped" is taking the side of the source of the risk.
* mine is about 10%: the Peter principle means they will always be used just above their competence, their competence is increasing, these combined means the damage they do when they fail is constantly increasing; fortunately, the rate of increase leaves room for governments to step in between "1000 dead" and "10 million dead", the former being "foolish boss used it to manage a chemical plant", the latter being "foolish boss used it for wet lab biological research" or "foolish politician used it for military planning".
I estimate only a 10% risk that the AI's competence rises to the level that it is given control over systems where failure kills everyone, rather than that governments agree to ban further development and limit usage.
My estimate of the risk of it causing a wet lab leak and triggering another global pandemic is much higher than 10%, but that is non-existential.