Exactly why we need causal reasoning/causal proof alongwith ML
That said in broader medical cases of ML where particular symptoms and measurements drive a diagnosis for example, I tend to agree.
If these models produce novel but (somewhat) effective structures, it must be because they pick up on less obvious patterns in the data. To be able to describe these would seem to super effective.