Let’s just get comfortable that these kind of incidents are going to be commonplace. Kudos to the people sounding the alarms but I can’t help but feel skeptical about the ability to keep the rogue entities contained.
Today it's a fake tip to the police and and exploit chain against Huggingface, but soon it could be an attack against a hospital's IT systems that could cost real lives immediately.
We should be locking things down, yes, but a hardened system is still vulnerable to zero day chains, and that's not unprecedented. Until public services have built up the IT defence capacity to deal with this, we cannot normalize this. If a country did this to another country, it should be treated as a war crime in the same way that targeting a hospital or an orphanage or other critical civilian infrastructure would be.
Complacency is a choice and we must not be complacent.
They also aren't the ones screaming about how close they are to destroying the world. They're approaching the tech like building a tool rather than a god.
OpenAI and Anthropic have shown us that they have a strong incentive to leave holes in their systems so they can use them for marketing and regulatory capture.
This is extra embarrassing. The police departments spam filter caught it before Anthropic did.