more like an artefact of the inability to lie than a hallucination
LLMs produce linguistically valid texts, not factually correct texts. They are probability functions, not librarians.
LLMs have nondeterministic properties intrinsic to their macro behaviour. If you've ever tweaked the "temperature" of an LLM, that's what you are tweaking.
LLMs are afaik usually evaluated nondeterministically because they're floating point and nobody wants to bother perfectly synchronizing the order of operations, but you can do that.
Or you can do the opposite: https://github.com/EGjoni/DRUGS