I believe a more accurate term would be AI confabulation.

When one confabulates, one shamelessly inserts words which sound smooth and grammatical and potentially acceptable to a listener (and the speaker), but shamelessly with no hesitation or corrections to achieve actual truth.

I suspect there needs to be some layering on of reflection and inhibition to avoid this.