> by what mechanism is an LLM going to help you design and manufacture a novel bioweapon? We know they cannot do spatial reasoning, and if they were this good at bio stuff then drug research would be absolutely out of control.

I guess my question would be how good the LLM is at writing a program that is good at spatial reasoning? If it can create a valid test suite from existing art on the subject it might be able to bumble its way to something specialized that it can then direct at a higher level.

Isn't it that kind of 'rapid self-improvement' that is the concern expressed in the linked statement?