Machine learning systems are hard because your system can be badly under performing but still doing objectively great. I've seen systems that produce amazing improvements in core business metrics and then years later some subtle bug in the labeling process is accidentally uncovered and fixing it boosts performance by an additional 20%.

A ton of time in ML is spent on minimizing the surface area for bugs because it can be difficult to even know they exist.