From one of the canonical papers on adversarial images:
> In addition, the specific nature of these perturbations is not a random artifact of learning: the same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input.
> ... suggest that adversarial examples are somewhat universal and not just the results of overfitting to a particular model or to the specific selection of the training set
https://arxiv.org/abs/1312.6199
More recent research shows that one can make adversarial images which fool ensembles. Once you have these images there is a very high chance it will also fool any individual architecture you pass it on to next.
https://twitter.com/goodfellow_ian/status/790330213963923456
These attacks show the inherent flaws of ML models (not just convnets, as per http://karpathy.github.io/2015/03/30/breaking-convnets/ ), and as far as I know, they are not trivial to work around. Research on adversarial images is still very active, in part because the military would like to rely on deep learning, and does not like to mistake a tank for a puppy (and vice versa).