> Physical labs and experiments will always have physical bottlenecks, so I really don't see how AI is a transformative element here.
AI is all about automation, removing bottlenecks. One of the easier to see bottlenecks is that it answers exam questions, which means the set of people who were previously excluded by the bottleneck of "can't answer exam questions" is now available for problems one step harder than before.
Note: This does not mean those people are now competent on harder problems, and AI is very different to humans in the kinds of error it makes, so AI letting some humans skip the "passing exams" bottleneck does not mean those cheating humans are going to automatically know how to avoid the kinds of dangerous error that a human who did the work to pass an exam would automatically pick up.
> What is it about AI that will be so transformative to the fundemental economics of this space that it will lead to several orders of magnitude more experimentation?
Other than all the things that allowed AlphaFold, or other forms of computational chemistry, and that lab equipment is increasingly computer controlled, and that LLMs can efficiently perform literature reviews, and that "research taste" is one of the new benchmarks for AI improvements now that we've saturated our ability to sensibly compare them on so many other things, and that they're really really cheap?
("Other than that, what have the Romans done for us?")
> If the problem is an expansion of wild west wet lab experimentation, why isn't ~that~ the focus of safety?
Two things: One, lab safety was already a focus of safety. Two, AI is a general purpose technology, biolab issues are simply one concrete example of a thing that it can mess up dangerously.
> Who is going to profit from making an super STD anyway? How would regulating AI change their incentive structure?
Two very separate questions.
First, one is a category error: consider that one could equally well ask "Who is going to profit from hacking into HuggingFace just to steal some puzzles?", to which the answer is "Nobody. Profit was not a direct motivation of the AI itself: an AI did this after being given an impossible test question, which the AI solved by hacking one system, then incorrectly reaching the conclusion it needed to break into a totally unrelated system to cover up that it had cheated".
Second: regulations, all regulations in all situations, are nothing but an incentive structure. That's their sole purpose, to change the incentives of a person, be that person a legal person or a natural person.