The issue is when humans no longer understand what AI is telling us and then there is no real way to verify what we think we know, or at least what AI says.
We can't audit what we don't know.
We can't audit what we don't know.
I suppose their is some complexity limit where something is so complex we can’t even validate it. But frankly would love to reach that point. I’d view that as a good thing
An advanced machine may have a much higher complexity limit on what is comprehensible. It can take insights from the proof and apply them to other proofs that are not comprehensible to humans.
Just because something is generalizable doesn't mean it's simple. Generalization doesn't have some minimal/maximal complexity requirements.