Failure would rarely be "catastrophic" in the context of web search as described in the article. If the ML model makes an incorrect prediction on new data you have a bad search result. No big deal - just feed the new data back into the model and learn again.

If the data set is large enough then the ML model may find patterns that escape a human expert. When it comes to finding patterns in very large datasets machines scale much better than humans. Given a large enough dataset a ML approach should be less susceptible to the Black Swan phenomena than human experts.

On the other hand, if failure of the system really could be considered catastrophic then there could always be a human involved. In these cases output from ML models could be one of the inputs that the experts considers before coming to a final decision. E.g. you wouldn't want a ML model doing medical diagnosis by itself but it could be very useful for identifying patients that should be double-checked, scanning diagnosis for errors etc.