Doesn't this assume the AI internally models the problem in a way that isn't foreign to the human? I believe the most effective AI will definitely not model problems like a human.
Isn't it more likely that we would look at a foreign model and think, "this makes sense, but i wouldnt have ever thought to do it this way".
Our senses might be fooled, but I think there would almost always exist a set of words to explain what the AI is doing assuming the AI is capable enough to find them
Thankfully, language is flexible so it can express many models we do not understand in ways that can help us understand them. It would be interesting to see some sort of study on where the boundaries are on what human language is capable of modelling or not