It's security research. You should want to know about all avenues of exploit regardless of intent. It's the only way to know risk.
Cache based side channels were in vogue over a decade ago. It was a point of contention back when people were first switching to AWS because "How can I be sure an attacker running on the same box as my service can't affect me?" is a big question that AWS needed to answer.
I.e. do not do these frontier model training runs on virtual machines in the cloud which depend on software systems to prevent internet access. Do them in a special data center that is physically disconnected from the outside world. Like a data center in a faraday cage under a mountain powered by a generator with no cables coming in or out.
I think we're learning the standards for sandboxxing change when you consider AI's ease in discovering exploits.
A sandbox good enough to contain human generated code and intent, is likely not good enough for AI generated code and intent (seems obvious to say that but).
And so poking on these issues, regardless of who asked to escape the box, is still worthy research.
- BSL4 is really, really stringent, and these requirements and review processes are strictly regulated across the board
- Most modifications researchers are actually capable of, nature is capable of producing something way scarier. Researchers are making tiny tweaks in comparison to the zoonotic exposure in everyday life that has brought us viruses that do things we never imagined
- Most gain of function tweaks weaken some other aspect, for example the way contagiousness and lethality are at odds
- It's considered rather unlikely that COVID-19 came about via gain of function research, despite how much the public wants this to be true
- It is quite useful for research, a lot is learned by doing it
I'm not even necessarily against GoF research, and I mostly agree with all of your points.
I'm asking about GP's logic.
I think different standards can apply.
You should want to know avenues of exploit so you can be defensive with security. You have little control over the offensive side of vulnerability exploitation.
Both lines of research are limited by some sense of waste: in time or money.
When should you stop? When some acceptable standard is reached and a certain amount of risk is accepted.
In the case of GoF, if it wasn't somehow profitable, then the research would likely cease to exist.
Also, you can't stop open source training so dealing with capabilities is better than trying to understand intent.
There isn't much history or stats when it comes to this kind of research.