We want our models to be opaque in the same way we like to ask people why they made a certain decision or act in the way they do. What's interesting to me is if you want to know about a person you'll get better information by asking their closest friends than themselves.

Perhaps there's something in the black box nature of our own self understanding similar to these hyper complex function approximators.

Judges make harsher rulings when they're hungry. A regulator might decline a loan and justify it b/c they had a bad experience with someone that reminds them of the person they're dealing with. AI makes bad decisions in boundary conditions or when they're trained on biased data. I think it's good to strive for opacity in every case but in complex decision spaces I'm not sure if we'll ever get fully satisfactory explanations.

Which is a meandering way of making the non-point that models making judgments have made me re-evaluate what I trust and why, and I think in many cases I'd rather trust a black box if I know what's gone into it and what's come out.