It more like asking the US government to get involved and if one of the 20 kids keeps on eating marshmallows, large numbers of federal agents raid the kid's offices and shut down the kid's compute resources.
US gov has direct jurisdiction over 4% of the worlds population and some pretty decent indirect influence over the other 96%.
It's good, but nowhere enough to prevent secret marshmallow eating on a global scale. Not even close
But even if that were not true, as a US citizen, even if there is nothing I can do about my getting killing by, e.g., Chinese AI researchers, I'm still going to work hard to prevent the AI researchers living in my own country from ending the human race. I'm responsible for doing my part in the governance of my own country, they told me in high school.
I see no good solution to this problem, no path to safety, but that does not mean I am not going to do what I can.
This isn’t about “dangerous” LLMs.
This is about unfettered LLM in the hands of the unwashed masses that actually tell the truth about what they find in their training datasets…
Also, the author (Eliezer Yudkowsky) calling for the shutdown of AI research on huge server farms doesn't have any stake (investment or employment relationship) in any AI company that would be harmed by telling the truth about what they find in their training datasets.