For example, if the AIs could design a complicated building, using our level of technology, they could explain to us how to build that building. And we could build it. Whereas if we come up with a better cockroach-house design, we cannot communicate it to the cockroaches, we simply cannot give them the information. So the AI->us is a very different relationship from us->cockroach.
This doesn't preclude that there might be some things the AI cannot explain to us. Only that there will be many things (infinite in fact) which they can explain to us.
I always thought you could ask GPT to illustrate the steps it took to arrive at the answer. I mean it can take your through the process it went through to arrive at the answer. Its as close you get to an explanation.