To extend the software analogy, asking mathematicians to decipher LLM-generated math results is like asking programmers to understand computer programs using a decompiler instead of starting from human-comprehensible source code.
People are confusing LLMs with a proof searchers. Yes LLM might be a bit like search but in the end it might explain every part.