This is one aspect about machine learning models I keep discussing with non-technical passengers of the AI-hype-train: They are (in their current form) unsitable for applications where correctness is absolutely critical.
The goal would be to find a way to make ML extrapolate only pixels that really describe actual really present features and never imagining detail that wasn't there in the first place. Now I am no expert at the matter, but what I know of deep learning models they are really good at the latter as they basically make statistic guesses on what would be plausible.
Getting a plausible guess on what looks like a convincing answer works really well for answering a question. But the problem at hand is more like predicting the words someone said based on the first and last word in a sentence. Imagine a criminal case where the evidence is fragmented like that: I am pretty sure a LLM could give a convincing prediction here, but I am not sure how much you could rely on that prediction being reflective of what was actually said. I certainly wouldn't feel comfortable with a conviction the result of that prediction even if it was reflective of the ground truth in 90% of times.
I don't wanna hurt anybodies feelings by stating that AI isn't a magical wand that makes everything better — but every technology has use cases at which it excels (e.g. pattern recognition) and use cases for which it is fundamentally unsuitable. If you try to screw on a nut using a hammer, that doesn't mean hammers suck, it means the user has a wrong idea what a hammer is capable of.
The point is: Don't be that person if you can avoid it.