That's not true at all. A language model can spit out a proof very similar to what a human might write. It might need some clean-up, but so do human-written proofs, actually.

People are confusing LLMs with a proof searchers. Yes LLM might be a bit like search but in the end it might explain every part.