The article was kind of interesting to me because it reveals that networks are probably sometimes making decisions on highly discriminating but non-essential stimulus features.
It's like the networks might have a high success rate with large number of replicated examples, over sample size, but not over a large number of distinct examples.
They're overfitting, but in a way that isn't immediately obvious because the test stimuli tend to be limited.
My guess is the answer is to incorporate into the training set a lot of quasi-random or structured but abstract images as controls for training.