That depends on two things:
1. How possible it is to inspect and criticize the judgements of the model, and its basis for it
2. How possible it is to inspect and criticize the judgements of the humans, and their basis for it
I would say that 2 is the bigger problem all in all. But 1 can potentially still become a big problem if models are trusted blindly.