This paragraph scares me. I'm not convinced that technology getting good enough to where it doesn't need to be understood to be used is a good thing in most cases. Especially when it's a black box.
Banking computer closed your account. Why? We can't tell you why.
Google or Apple rejected your app from their app store. Why, won't tell you why.
Me trying to get my prescription refilled at Walgreens. Walgreens computer shows they never got it from the doctor. Doctor sent it 5 times. Turned out the computer was marking it invalid each time. Pharmacist isn't allowed to tell neither me or my doctor why.
And it doesn't take a deep CNN to become a black box - any sufficient complex algorithm will do. I doubt anybody in the world understands all the details of how the latest CPU actually works, for example.
You would not believe a human is intelligent if he just tells you what to do without being able to explain why, would you?
Or think of the “uncanny valley” in 3D graphics -- most people can tell when a rendered image looks slightly off somehow, but most people can’t pinpoint the exact problems. “Something about the eyes just isn’t quite right”, etc.
But experience people can. I've worked at a 3D animation studio and we had this 'old guy' who started his career as a classically trained artist and sculptor. He could look at a character model that we all agreed felt a bit 'off' and point out that the problem is how you'd modeled the muscles connecting the shoulders to the arm. Hell sometimes he'd look at a model that we all felt was fine and suggest tweaking the bridge of the nose and that made it much better.
Will it matter whether the AI is "really" intelligent when companies that implement the black box to spec thrive, while the companies that insist on having a human understand everything fall behind and fail?