If someone asks the model how to create a pandemic I think it would be pretty bad if it expertly walked them through the steps (including how to trick biology-for-hire companies into doing the hard parts for them).
What is far more likely is that the development team will build a model that often mistakes legitimate use for nefarious intent while at the same time failing to prevent a tenacious nefarious user from getting the model to do what they want.
(Possibly, though, this is worth it on balance as a kind of practice? If they can't even keep their models from telling you how to hotwire a car when you ask for a bedtime story like your car-hotwiring grandma used to tell, then they probably also can't keep it from disclosing actual information hazards.)
Another time I tried to let it generate a very specific Sci-fi helmet which covers the nose but not the mouth. When it continusly left the nose visible, I tried to tell it to make this particular section similar to Robocop, which caused it again to deny to render because it was immediately concerned about copyright. While I at least partially understand the concern for the last request, this all adds up to making this software very frustrating to use.
Long term limiting LLMs isn't a solution, but while we get the laws and practices around risky biology into better shape I don't see how else we avoid engineered pandemics in the meantime.
(I'm putting my money where my mouth is: I left my bigtech job to work on detecting engineered pathogens.)
This particular hole is not original to me, and is reasonably well known. A group trying to tackle it from a technical perspective is https://securedna.org, trying to make it easier for companies to do the right thing. I'm pretty sure there are also groups trying to change policy here, though I know less about that.
In justifying your post, you actually answered contrary to your original assertion. The information is out there, we should talk about it to get the issue fixed. The same justification applies to avoiding LLM censorship.
There's a sea-change afoot, and having these models in the hands of a very few corporations, aligned to the interests of those corporations and not individuals, is a disaster in the making. Imagine the world in two years... The bulk of the internet will be served up through an AI agent buffer. That'll be the go-to interface. Web pages are soooo last decade.
When that happens, the people controlling the agents control what you see, hear, and say in the digital realm. Who should control the alignment of those models? It's for sure not OpenAI, Microsoft, Google, Meta, or Apple.