Not saying they aren't real risks... Are there going to be sad incidents where innocent people will be harmed? Yes. The question though is if there will be enough of them, and egregious enough, for us to disregard the upsides of technology that is not heavily regulated.
Regulations won't fix a damn thing and regulators have no chance of keeping up even if they wanted to. We need these tools to not be developed or offered at all by choice. Unfortunately that really means either the company, investors, or customers need to make a moral decision at the cost of economic gain or convenience/novelty.
To be clear, the context of this thread is extremely important. I don't know how far our existing laws might go to make sure ML generated child porn is illegal and I'd personally feel much more comfortable to know that ML generation isn't some kind of legal loophole.
I assume that training separate models for separate jurisdictions would be rather expensive, so probably unlikely.