Once you can run decent image generation models on your phone, what shall we do? Prohibit apps that allow you to run custom models? Add a mandatory safety check?
It seems like a pointless exercise to me - better punish the actual crime rather than try to regulate the tech.