kind of boggles me that the big ai labs arent running zero-shot classification on all their outputs with "does this look like any of these bad things we know people get up to"
They do. They're getting increasingly cautious with these things, too- Claude won't help you reverse engineer much anymore apparently.
Hackers are almost certainly using open-source AI for this.
Oh, other day I was asking Claude to get a onetime code from my PocketID VM, I was being lazy and I was asking it to open the website, generate the authentication code from there and then type it into a built in browser. And it just refused to do it. I couldn't convince it to do it, even though it was all my own infrastructure And essentially, we were doing a security theater because the website that it was trying to authenticate to was the one it had just published itself behind my internal trafefic instance, lol.
I ended up having Codex do it. Codex had no compunctions about it.
They certainly do.
Just not very well. It's a hard problem. And they're not incentivized to slow down.
they are, human post training is a pretty big and growing industry and is exactly like that