Well, it could. There are really capable self-hostable models.
MTPE, "Machine Translation Post Editing", is what has become the norm.
AI generates your first draft. Humans post-edit the output as a final draft.
I imagine most AI use cases will still have a human in the loop for quality assurance. (The goal of AI doesn't need to be 100% accurate as long as the first draft is able to be post-edited and reviewed by a human who ultimately takes responsibility for the output - assuming post-editing/QA takes less time than writing the first draft yourself)
I think that simple rubber-stamping would happen in any situation where the input was 'good enough' most of the time. And so the Bad Things and hallucinations would still get through
It all comes down to quality at the end of the day. The person doing the work will be fired if the quality of their output isn’t up to standard, assisted by AI or not. That’s how the language translation industry has operated successfully for years.
Completely agree the natural instinct is to rubber stamp. But in the language industry, their boss looks at metrics like “percent of translations edited” and translators reviewing machine translations will get flagged and loose the work if they are bypassing the expectations and not doing the job.
In other words this is a mostly solved problem. Put the responsibility on the worker for the AI’s output, and the worker will care as much about its output as they care about their job. Which also applies in the software field. Employers are generally fine with Copilot, etc, but it’s not an excuse for sh*t code. That same model can be applied in different contexts.