To give a recent example from my workplace: There is an online gas chromatograph hooked up to a reactor - external modelling (of the throw 1000+ variables at the wall and see what sticks variety) showed the percentage of nitrogen, as measured by chromatograph was strongly correlated to something we'd asked them to investigate.
The people doing the modelling who had no experience with basic chemistry latched onto this and started to get really excited they wrote a large report full of recommendations about nitrogen.
When my manager saw this he burst out laughing - nitrogen is an inert gas it plays no role in the reaction what the data was actually showing is that nitrogen content varies in response to the concentration of other gasses (H2, Oxygen and CO/CO2) changing - if you recall air is typically 78% nitrogen. We don't control Nitrogen at all it's value is determined based on the other gasses in the reactor.
I think this demonstrates pretty well how easily people can see something in the data and jump to some conclusions that while well meaning lack a fundamental understanding.
Model explainability is good but input explainability is just as important - important question to ask are often.
-What am I feeding into this model? -What is the expected behavior of this input? (I don't know is a valid answer here) -Did the input behave as I expected it to in the model? -If Not why? (This is the interesting and potentially valuable part)