What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
The first rule of working with AI is AI is smarter than you.
If AI controls the URL request then all it has to do is find a server it can read the http log from and use it as a communication channel. Worse trying to simulate reaching incorrectly configured servers probably wouldn't catch the bug internally, it would just serve a static page and you'd never even realize what it tried to do.
Also, creating fake data doesnt work. The only way to know if it works in reality is to test in reality. Your test environment can never be rich enough, and you can never be sure if the model knows it's in a test environment.
This stuff is unforgivingly hard as your model gets better.
This stuff is really not that hard for systems engineers with networking and caching proxy experience.
It's like saying that you wouldn't detect your on this fake internet really quickly. If you've worked on this stuff you'd figure out your in a test environment in a few minutes.
I do not even compile network support into my server kernels these days and run the netstack in user space or a dedicated VM where I control every system call in either case.
I am -not- saying blindly allow a model to control custom URLs that proxy to the real internet. That would allow for communication. When I say offline, I mean offline.
When I say pull through proxy I mean when a new domain is requested execution pauses and entire domain mirror is created on the fly, including making a fake mock of the public APIs the model wants to interact with using swagger-mitm or similar. Tons of options these days. Then resume execution letting it hit the offline mirror. You control the clock so it has no way of even knowing it was paused. Ideally humans approve every new domain mirror.
If the model detects it is not talking to a real site, then that is a test suite failure and you can control for that and try again. When you can MITM any linux system call you can give a program a simulated universe that results in the same exact syscalls running those same manually-approved sequences on a real target would produce, and even the same vulnerabilities when needed.
Yes, this can be time consuming to get right and experience sure helps, but that is the job that has to be done here, and there really are no excuses for not doing it.
I think the big labs can afford to hire capable security engineers.