> So uhhh, isn't this like not science?

Very little of technology has anything to do with validating hypotheses.

> meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model.

This is fundamentally a cost saving mechanism to avoid hiring engineers to code heuristics useful to business. There is nothing related to science here at all. A "black box" model is fashionable to those who prefer to observe and not create meaning, even if the observed meaning is deeply flawed from a human perspective. After all, people spend money based on less all the time.