We do have a "working hypothesis" how the mind works though, what psychology calls "theory of mind" or "folk psychology". While this is completely divorced from the level of neurons or brain regions and so might tell little of how the mind "really" works, it's detailed and useful enough that we can judge how other people would react, guess what attitudes, feelings and motivations drive them and in general can have a useful level of interaction.
The same is true for traditional computer programs. The standard argument against valuing understanding is "well, you don't understand the assembly, processor microcode and the electrical signaling when your program runs either, why do you care about understanding the high-level code then?"
I'd counter this by saying that high-level languages have (mostly!) self-contained abstractions that allow you to "understand" a program deep enough to make accurate predictions of its runtime behavior, answer "why" questions when errors happening and safely make changes to it - without having to understand all the stuff at lower levels.
In contrast, with AI, I see the risk that we lose understanding of both the low- and the high-level structure of things. Then we really end up in "unpredictable alien artifact" territory.