IMO a part of the problem here is that you have a misunderstanding on the part of deep learning people. They look at radiology, and they say "these people are just interpreting these pictures, we can train a deep learning model to do that better".

Maybe there's a bit of arrogance too, this idea that deep learning can surpass human performance in every field with enough data. That may be the case, but not if you fundamentally misunderstood the problem that needs to be solved, and the data you need to solve radiology, for instance, isn't all in the image.

Somewhat related: another area where DL seems to fail is anything that requires causal reasoning. The progress in robotics, for instance, hasn't been all that great. People will use DL for perception, but so far, using deep reinforcement learning for control only makes sense for really simple problems such as balancing your robot. When it comes to actually controlling what the robot is going to do next at a high level, people still write rules as programming code.

In terms of radiology and causal reasoning, you could imagine that if you added extra information that allows the model to deduce "this can't be a cancerous tumor because we've performed this other test", you would want your software to make that diagnosis reliably. You can't have it misdiagnose when the tumor is on the right side of the ribcage 30% of the time because there wasn't enough training data where that other test was performed. Strange failure modes like that are unacceptable.