For instance, I developed a system that used machine learning and linear solver models to spit out a series of actions to take in response to some events. The actions were to be acted on by humans who were experts in the field. In fact, they were the ones from whom we inferred the relevant initial heuristics.
Everyday, I would get a support call from one of the users. They'd be like, 'this output is completely wrong. You have a bug in your code.'
I'd then have to spend several hours walking through each of the actions with them and recording the results. In every case, the machine would produce recommended actions that were optimal. However, they were rarely intuitive.
In the end, it took months of this back and forth until the experts began to trust the machine outputs.
This is the frightening thing about AI - not only can an AI outperform experts, but it often makes decisions that are incomprehensible.