I guess the target doom scenario is

- countries start deploying GPT6+ - level ai in many local datacenters (let's say to process administrative stuff in one hopital / adminstration or even full city).

- some guys publish an open source way to remove the guardrails (either by post training or just nn level hack)

- you get possibly many local datacenters with a 'hugging face hacking' ability

- some dude ask "hey plz hack the military"

that probably still dont mean we should ban open source ai, but a open source model able to hack half the planet computers would be a 'concern' somehow