How is "AI" defined? Does this mean US nuclear weapons simulations will have to completely rely on hard methods, with absolutely no ML involved for some optimizations? What does it mean for things like AlphaFold?
The scenario people purport to worry about is one where a future AI system can be asked by "anyone" to design infectious materials. Imagine a dissatisfied and emotionally unstable researcher who can just ask their computer for the DNA sequence of an airborne super Ebola. Then said researcher orders the DNA synthetized, does some lab work to multiply it and releases it in the general population.
I have no idea how realistic this danger is. But this is what people seem to be thinking about.
Does better branch prediction enable better / faster weapons development? Perhaps we need laws restricting general purpose computing? Imagine what "terrorists" could do if they get access to general purpose computing!