How exactly does your brain do route planning?
How do you pick which lane to turn down in the parking lot when looking for a space?
Why did you get off at exit 14?
Wait, this isn’t even your turn, why did you go left here?
I mean, when you get down to it, why are you even driving to this deadend job?
But yeah, sure, humans can ‘explain’ their behavior, which is why we can trust them.
That is not the kind of explanation that is needed in AI systems. When people talk about "explainable AI", they literally just mean systems that can answer the kind of question that a human would be able to answer.
That's because a question that a human cannot answer is very likely to have an answer that a human will either not be able to understand, or will have to work very hard to understand... which is no better than no explanation
But when an AI gets sufficiently complex of course there won’t be explanations that make sense for those kinds of errors, because just like a human the AI is integrating lots of different bits of information that it has learned are important and it has limited capacity and sometimes it just gets its attention focused on the wrong thing and it just didn’t see the guy, okay?
Demanding that AI be explainable is fundamentally demanding that it not be intelligent.
That out of the way, there are AI approaches that can explain their actions just fine without going dumb. For example, I posted this comment earlier:
https://news.ycombinator.com/item?id=30872400
about an AI system called Nook that recently won a tournament against 8 human champions of the card game Bridge. In Bridge, players must be able to explain their moves, so an AI player without the ability to explain its decisions can't play a full game of Bridge.