This reminds me of the Is-Ought problem defined by Hume. "Ought" has to do with value judgement, and that is squarely in the realm of human morality. Only humans know what humans value; generative AI can't replace that and it would be an enormous mistake to try to do so.