We as humans are very good at coming up with solutions to ambiguous/messy real world scenarios... robots less so. By definition, the current set of robots are programmed for specific situations.
As we automate more and more, I suspect we will have more and more “one off” catastrophic events. Yes, once we experience them, we can “test” against them. However it almost seems to argue for one amount of human in the loop no matter what, if only to give an occasional “that seems weird” check.
Am I missing something? Yes we can make an argument for machine learning to learn “nominal” and report off nominal... but that only works if you gave the right sensors being inputted in the first place.