Research identifies key weakness in modern computer vision systems
news.brown.edu
news.brown.edu
1) It's not published in a good CV journal.
2) It seems quite likely that they've just identified the problems described and fixed by [0, 1] (of which [0] is the much better paper).
In brief, CNNs guarantee translation invariance (ignoring boundary effects) through translation equivariance of their feature maps; this borks certain problems, most importantly a commonly used mechanism for instance segmentation as described in [0]. Consider trying to instance-segment two copy/pasted faces; translation equivariance and global pooling clearly make it impossible.
I strongly suspect that the problem described in the paper originates from a similar place, and thus could be solved in a week or so. I doubt it's a "key weakness of modern cv systems".
[0] https://arxiv.org/abs/1807.10712 [1] https://arxiv.org/abs/1807.03247
EDIT - a little further investigation shows that it was accepted as a workshop paper at ICLR, also provides the reviewers comments. https://openreview.net/forum?id=HymuJz-A-
So perhaps I'm being overly negative. Can't find any code release for it, would be interested if anyone can.
EDIT2 - Entertainingly, the workshop paper version of this and the CoordConv paper seem to use the same name for different toy tasks, "Not-so-CLEVR".
"However, I think there's some chance that if the same tasks were in the hands of people who wanted CNNs or CNN+RN to work well, the results might have been different."
The reviewer goes on to write that he cannot say what people with the proper attitude would do differently.
I guess that's an undesirable symptom of CNNs (etc.) still being an empirical rather than theoretical science.
Many networks can detect if two never-before-seen objects are the same or different. Simple example is Koch's Siamese Neural Networks[1]. More useful example is Facenet[2], which can tell if two faces are the same or different, even if never trained on these specific faces. Facenet learns an n-dimensional embedding and similarity is estimated by euclidean distance. These are cool papers by the way. The Omniglot dataset[3] used in Koch is a sort of an "inverse MNIST". Omniglot has 20 examples each of 1600 different characters, vs MNIST's 60,000 examples of 10 different characters.
[1] https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf
It should be possible to do negative analysis of AI capabilities. But since there is almost no theory behind this stuff right now (just test sets and architectures targeting particular test sets), anyone can always claim that any negative analysis just didn't target the "right" architecture.
Faces have a limited set of common features that are readily separable, quantifiable, and comparable, which is why face recognition can be done rather well just using a SVM. When the set of common features falls to zero, as in this paper, a face-specific net architecture will not do one-shot recognition across independent faceted objects nearly as well.
In any case, they directly exclude the case you mention.
Their paper was still unimaginative. Trying a handful of vanilla networks doesn’t prove anything.
The right framing would have been, “here’s an interesting case to work on” instead of pretending to have identified a basic limitation of neural networks. This is more like a research idea than research.
To be honest, I skipped it and went to the paper :) Apologies!
There is a lot of research on attention and saliency mechanisms. Itti, Koch, Niebur had a model. Adaptive resonance theory from Grossberg has a successive winner take all strategy as well. Global workspace theory from Baars similarly. It is a very logical cognitive architecture to come up with. If I'm correct also the new architecture from Hinton separates object recognition from the position information. That these are separate pathways in mammals is also already long known.
It would have been nice if they would have come up with a method in which these separate pathways and/or a sequential attention mechanism would arise from more basic principles and not hardcoded by researchers.
“When computers look at an image, they can’t actually tell where one object in the image stops”.
“When children read French, they actually don’t know what the foreign words mean.”
Just means: Todo: teach them to.
The proper solution can be implemented as following. Firstly either using sliding window or selective search like algorithm to generate potential object targets, secondly using trained object models on these targets.
In my opinion, it seems that the researchers didn't know how to use cnn.
There's only one problem in computer vision, and it's not unique to the field at all: balancing sensitivity vs specificity. Modern applications of ML don't solve that.
The examples used in the article (in some cases the objects were identical; sometimes they were the same but with one object rotated in relation to the other) are solved very easily with OpenCV "template matching".
It's much more effective and economical to build a simple hybrid which knows when to use CNNs vs other methods - and ensemble the results.
When you know that you have a problem at hand, where template matching would be a better approach, you can use that. However, it could be that more complex tasks (where usually CNNs are used) for which it's not immediately obvious that this capability is required, would also benefit from this capability.
Ensembles are a practical solution, but it's kind of ugly. Remember, this is research into limitations and capabilities, not about solving particular tasks. Once the issue is identified it can be tackled and the resulting model may be nicer and more elegant than an ensemble.
if the answer is no, then I don’t think that even feedback would help in the original problem..