At some point it will be less wrong (TM) and it'll be helpful. Feels generally like a good bet.
Fundamentally this is an alignment problem.
There isnt a single AI out there that wont lie to your face, reinterpret your prompt, or just decide to ignore your prompt.
When they try to write a doc based off code, there is nothing you can do to prevent them from making up a load of nonsense and pretending it is thoroughly validated.
Do we have any reason to believe alignment will be solved any time soon?
This isnt a matter of training data quality.
If we could align an AI with incentives in the same way we can a person then youd have a point.
So far alignment research is hitting dead ends no matter what fake incentives we try to feed an AI.