I read about a case here on Hacker News a while ago – I can't seem to find the link again – where the police arrested a guy for committing a crime because they had ran a face detection model over drivers license pictures and CCTV footage and it flagged the guy. Problem was – he didn't look like the perpetrator at all, it was clearly just a fluke. But since the police was not trained to scrutinize the face detection results, they arrested him and had him in custody for a good while.
Another thing that gets more and more common is that when traveling in a foreign country and using a foreign credit card, transactions at tourist-heavy businesses like restaurants or rental car companies go through smoothly. But try to order a T-shirt online? Oh no, that must be fraud! It's constantly blocked, and always on the merchant side, my bank knows about my traveling. Turns out, many fraudulent transactions are made with stolen, foreign cards, and so a primitively trained model might look successful enough if it simlply blocks all transactions with foreign cards.
But of course it could have also been some other mistake that lead to this. ML is a good possibility though.