As with a lot of AI stuff, Borges already wrote about it.
https://www.sccs.swarthmore.edu/users/08/bblonder/phys120/do...
https://www.sccs.swarthmore.edu/users/08/bblonder/phys120/do...
Btw I am not disagreeing with the utility of LLMs, my point is it can never be 100% accurate with current architecture (unless you blow up the size).