I think that's unavoidable. There is always a level at which your understanding of your tools gets fuzzy and incomplete. Do you understand everything about what your compiler is doing? Your OS? Your could provider? The hardware any of them are running on? The majority of us are standing on the shoulders of the few that do understand those things and provide them to us to use.
And if you mean that we would not understand what the software we write is doing to produce the result it does, aren't we already there with machine learning?