>> ...AI systems whose actions cannot be closely scrutinized and explained...
> Author is one of the large group of people who are in for a shock
> when they try to scrutinize and explain a humans crazy actions. I
> can't even explain why I get the wrong result sometimes when I add
> numbers in my head.
I was just rereading Asimov's
I, Robot, and was struck by this passage:
"The cotton industry engages experienced buyers who purchase cotton. Their procedure is to pull a tuft of cotton out of a random bale of a lot. They will look at that tuft and feel it, tease it out, listen to the crackling, perhaps, as they do so, touch it with their tongue, and through this procedure they will determine the class of cotton the bales represent. There are about a dozen such classes. As a result of their decisions purchases are made at certain prices, blends are made in certain proportions. Now these buyers can not yet be replaced by the machine.
"Why not? Surely the data is not too complicated for it?
"Probably not, but what data is this you refer to? No textile chemist knows exactly what it is that the buyer tests when he feels a tuft of cotton. Presumably there's the average length of a thread, their feel, the extent and nature of their slickness, the way they hang together and so on. Several dozen items, subconsciously weighed out of years of experience. But the quantitative nature of these tests is not known. Maybe even the very nature of some of them is not known. So we have nothing to feed the machine. Nor can the buyers explain their own judgment. They can only say, 'Well, look at it. Can't you tell it's class such and such?'"
Asimov wrote this in the 1940's and in this passage he tried to illustrate how unquantifiable and impenetrable human judgment was compared to artificially intelligent robots, which he saw as ultimately rational calculating machines. Ironically, today AI techniques such as neural networks are criticized for making much the same sort of impenetrable judgments.