If we can't understand the problem, will we be able to understand the solution presented by that AI? Or we'd just apply it, trusting blindly the unfathomable reasons the AI used? Do we have an AI where the decision tree can be grasped by humans?
Top players routinely use them in preparation, to study new lines, get hints about what moves make sense in a position and also to generate new ideas or tweak the principles they apply in the game. The engines don't explain their reasoning, but provide something closer to the "correct" move in any given position. It's up to the humans to do the legwork and understand _why_ the recommended move is strong.
Clearly, chess is not real life, but the impact of these oracle engines has been broadly positive (with the exception of using engines to cheat in online play).
But when it comes to raw intellectual superiority, say, explain the logic behind new mathematical or physical concepts and discoveries, very high level predictions, we come to our human limit and cannot surpass it.
The AI can give us implants to improve our intelligence, but only to a certain limited degree, our biological brain will slow us down significantly.
That is the barrier we cannot possibly cross in a timespan of human life.
And to be honest, I wondered about that with GPT-3. Maybe it could give a more profound answer to a given prompt, but it chose not to, since it found the prompt to be too silly, and it responded in kind. Just like adults are able to entertain an imaginary universe of children.
So even to explain that we want a "serious" answer might be difficult.