I don't think it's the X-risk AI threat that should primarily concern everyone. I'm not suggesting it should be ignored; some like Scott and Eliezer are working on trying to reduce that risk (or more accurately, trying to figure out ways to understand the alignment problem well enough to develop ideas to reduce the risk), but it's out of the hands of most of us. Either AGI instantiates and kills (most) all of us, or it doesn't. And, as Scott points out, humans—due to various technologies and psychosocial problems—may be running headlong into a great filter in the near future without AGI, so an AGI might be an improvement in that case.

But what happens to society when all non-face-to-face communication becomes saturated with unlabelled AI-created and AI-assisted content, which may or may not be trustworthy or correct, but which we can't separate from human-created content? What happens to economies, governments, countries, and geopolitical stability? You don't have to believe in a large-scale AI replacement of human work and massively increased unemployment to see the problems AI is already creating, and LLMs will only get better.