As soon as we developed agentic, persistent reasoning workflows, the supposed harmlessness of language models went out the window. I don't know if that ends up being The Big One, but with how rapid progress still is, I'm not liking the odds right now. And we've got a trillion dollar army of coders ready to try and bypass any scaling barriers that do end up existing, eg with "World models".

I am not confident in humanity's ability to write harnesses that perfectly impede a superintelligence with 100% accuracy and reliability from pursuing its goal even in ways that humans categorically would not want it to. I am even less confident in restraints in areas where humans have actual incentive for conflict.