The human mind has been a black box, and largely still is.
So we traded one black box for another, in more than one way. The new black boxes are going to be able understand and keep up with their own work, much more quickly than we will.
The "alien" picnics, in this case, are a prelude to settlement and expansion, and we have only met the children.
Your point on understanding mostly holds (although our own mind's inscrutability but usefulness is a strong counter-case), but it will increasingly work against us.
The same is true for traditional computer programs. The standard argument against valuing understanding is "well, you don't understand the assembly, processor microcode and the electrical signaling when your program runs either, why do you care about understanding the high-level code then?"
I'd counter this by saying that high-level languages have (mostly!) self-contained abstractions that allow you to "understand" a program deep enough to make accurate predictions of its runtime behavior, answer "why" questions when errors happening and safely make changes to it - without having to understand all the stuff at lower levels.
In contrast, with AI, I see the risk that we lose understanding of both the low- and the high-level structure of things. Then we really end up in "unpredictable alien artifact" territory.
We have those concepts. We apply them every day as post-hoc interpretations - which is not how real theories work.
We could label this kind of stuff "human slop", but I think "bullshit" is the technical term.
We are a long ways from knowing how to manage humans in a way that prevents them from doing great damage to each other, even when the result is a disaster for themselves. Look in any prison or at many world leaders.
I'd argue the entirety of fictional characters works by this. It would be impossible to write a fictional character if you couldn't "simulate" their behavior in your head to some degree. (Some authors say they aren't even consciously thinking about this, they feel as if they just "observe" the characters and writing down what they see)
> We are a long ways from knowing how to manage humans in a way that prevents them from doing great damage to each other
Predicting and "managing" are two different things. Only because I know someone is an asshole doesn't mean I know how to change him to not be one.
You are equating plausible with true, familiarity with prediction. When it suits you, and then not when it doesn't.