Why am I supposed to believe that a virus created in a virology lab with AI assistance is uniquely more dangerous than a virus created in a virology lab without AI assistance?
You're not.*
The difference is how likely this is to be done, not how dangerous it might be if it was done.
And this isn't likelihood in a 0-100% sense, but in a Poisson distribution sense, i.e. mean time between incidents.
Remember: these AI minds aren't that good with the physical world, and have nevertheless been repeatedly connected to robots, and now some of the AI labs are showing off their actual wet labs. Accidents are absolutely a possibility, and I don't think it's low given the previous behaviour of Silicon Valley startups.
* OK, some people care that AI is getting more capable at genetics just like it's been in maths and programming, but short term, before AI solves genetics so hard it can make an STD that makes the infected uncontrollably horny and then ossifies our bodies or something, it can obviously just copy any of the many DNA or RNA sequences we've already got on record.
Who is going to profit from making an super STD anyway? How would regulating AI change their incentive structure?
AI is all about automation, removing bottlenecks. One of the easier to see bottlenecks is that it answers exam questions, which means the set of people who were previously excluded by the bottleneck of "can't answer exam questions" is now available for problems one step harder than before.
Note: This does not mean those people are now competent on harder problems, and AI is very different to humans in the kinds of error it makes, so AI letting some humans skip the "passing exams" bottleneck does not mean those cheating humans are going to automatically know how to avoid the kinds of dangerous error that a human who did the work to pass an exam would automatically pick up.
> What is it about AI that will be so transformative to the fundemental economics of this space that it will lead to several orders of magnitude more experimentation?
Other than all the things that allowed AlphaFold, or other forms of computational chemistry, and that lab equipment is increasingly computer controlled, and that LLMs can efficiently perform literature reviews, and that "research taste" is one of the new benchmarks for AI improvements now that we've saturated our ability to sensibly compare them on so many other things, and that they're really really cheap?
("Other than that, what have the Romans done for us?")
> If the problem is an expansion of wild west wet lab experimentation, why isn't ~that~ the focus of safety?
Two things: One, lab safety was already a focus of safety. Two, AI is a general purpose technology, biolab issues are simply one concrete example of a thing that it can mess up dangerously.
> Who is going to profit from making an super STD anyway? How would regulating AI change their incentive structure?
Two very separate questions.
First, one is a category error: consider that one could equally well ask "Who is going to profit from hacking into HuggingFace just to steal some puzzles?", to which the answer is "Nobody. Profit was not a direct motivation of the AI itself: an AI did this after being given an impossible test question, which the AI solved by hacking one system, then incorrectly reaching the conclusion it needed to break into a totally unrelated system to cover up that it had cheated".
Second: regulations, all regulations in all situations, are nothing but an incentive structure. That's their sole purpose, to change the incentives of a person, be that person a legal person or a natural person.
> lab safety was already a focus of safety.
Thank you for this insight.