IMO a sufficiently advanced AI should be able to do a full analysis of its neural architecture and explain it and break down its functionality.
(Just deciding that your LLM has magic powers because you've put it in a category called "AI" and you've decided that category has said magic powers is what that guy Wittgenstein was complaining about in philosophical problems. Besides, intelligence doesn't mean all your thoughts are automatically correct!)
This is the true bitter lesson for HN.