Until your model starts misdiagnosing people in real life because your training data wasn't 100% perfect (which it never will be for complex real-life problems).
The problem with unexplainable AI isn't just the lack of explanations. It's the fact that the entire model is a black box, so you cannot somehow tweak a single part of it without retraining.