This is the main risk that the labs OpenAI and Anthropic have called out for and asked for slowing the frontier.
Almost all people thought that this was fear mongering, hype marketing and regulation capture.
The same people will blame OpenAI and Anthropic for creating this.
I found this relevant in yesterday’s ML4 announcement:
> This is particularly important in cybersecurity, where provider-level refusals can block legitimate vulnerability research and incident response, and where losing access to a capability mid-incident can itself become a critical security risk. ML4 pairs top-tier cyber performance with open weights and self-deployment, giving organizations both the capability and the autonomy to run advanced security work under their own policies.
I prefer not to have oligarchs and governments deciding what prompts are allowed and which need to be modified before I get to see it.