I wonder if existing mipmap algorithms and hardware could be used for this kind of representation?
4,164 karma · joined October 23, 2007
This is very abstract, but more concretely, what I mean is that it helps to read about the philosophy/history of science. The canonical recommendation here is Thomas Kuhn's The Structure of Scientific Revolutions [1], but I also strongly recommend watching James Burke's Connections tv series [2] and then watching or reading his The Day the Universe Changed [3] to understand how our entire conception of "reality" or "truth" is so strongly determined by the technology of the time.
Finally, there are lots of "truths" in fields where this is no scientific basis to fall-back on -- politics, art, etc. For these, you have to seek out and find the best elucidators of each field (usually, but not always, books).
[1] http://en.wikipedia.org/wiki/The_Structure_of_Scientific_Rev...
[2] http://en.wikipedia.org/wiki/Connections_%28TV_series%29
[3]
Fortunately, OpenCV provides functions to train your own classifier using your own data, so if you can get good data, and are willing to wait for the week or two it'll take to process, you can get much better performance using the same technique.
Finding sufficient training data (labeled faces) is more tricky. For an automated approach, you could try bootstrapping it by simply running a more state-of-the-art detector (such as on Facebook, Google, etc.) and feedings the outputs into the training function (you'll need to do some work to actually get the outputs programmatically, perhaps through undocumented APIs).
Another approach is to use an existing face dataset. One possibility is the FDDB [1] although it's kind of small (5171 faces). Unfortunately, most other datasets I know of in the face community are either unrealistic images (e.g., only "mugshot"-style) or were detected using OpenCV (and thus won't help you get better).
The final approach is to label images yourself, which would require time and money (e.g., to do it on Amazon Mechanical Turk).
If you do decide to retrain classifiers, it's essential that you augment your data in various ways, such as by mirroring images (and labels) left/right, generating slightly rotated versions of faces, and at slightly different sizes. This augmentation can GREATLY improve the performance of your detector.
For example, you might get lucky and 2 of the 3 test photos you have match well with 2 of the 5 database photos of the right person. But if you have enough people in the database, you're almost guaranteed to find some random person where all 3 of the test photos match quite well. So it often comes down to how you combine the results from the different test images. Do you take the max? Do you average them? etc. Each has different tradeoffs, and it's a topic under research at the moment ("image set recognition"), but I'd say there are no compelling results yet.
Some personal devices might indeed be tuned for relatively fewer people, but these are precisely the cases that are the toughest: distinguishing between people in the same family, who're all going to look much more similar to each other than random people in a large database.
But there are many scenarios where it might actually be distinguishing between 1000s of people. To take but one example, advertising. On the web, advertising is popular not just because it's "easy" and one of the few ways of successfully monetizing a site, but also because knowing more about a person allows for more targeted advertising which has a much bigger ROI. How would this translate to the "real world?" Personalization is clearly one of the ways to do this, and face recognition is ideal in many respects, given that it works from a distance, is non-invasive, and cheap to implement. (Obviously, there's a separate debate to be had about privacy and other issues, but I think it's hard to deny that many companies are interested in this.)
I'm not sure that face recognition will be used for security purposes like house/car locks, just because it seems like other devices might get you better security sooner and cheaper (like RFID tags used in some car keys now, or some sort of ID from your phone).
If you have a ton of labeled data (either for the exact task you're doing, or a related one), DL often wins because it's sufficiently high-capacity to learn sophisticated models from all that data.
On the other hand, if you don't have enough data, then the deep network isn't going to be learnable, in which case hand-tuned methods might do better, depending on how much "domain expertise" you can bake into the system. Faces are a very constrained domain, and since there has been enormous work on them for decades now, it's one area that non-DL algorithms probably still have somewhat of a fighting chance (until people gather & label large enough databases, of course).
A computer algorithm operating on single 2d images (like LFW) has none of that. If you were to ask people to do the same task with the same data, they'd probably still do pretty well (much better than computers), but perhaps not perfectly.
The differences between image-to-image (same person) and different people is exactly what makes this problem so tough in the general case. It was shown about 20 years ago now that faces span a fairly low-dimensional manifold, and across different parts of this manifold, faces of different people do look much more similar than faces of the same person.
I probably shouldn't have explicitly written that formula for recognition, since it's not actually the formula, but more a general scaling rule-of-thumb. But even if we take it as given (hypothetically), there would be several issues. First, remember that 97.53% is on this particular dataset, which is very special in many ways (e.g., it was all collected over the course of a year from photos on Yahoo News, of public figures, with relatively lower-resolution images, and often very distinctive backgrounds). On a more realistic dataset, these numbers would be much lower.
Second, having more images of a person does help, but not nearly as much as you'd hope, because now there's an even bigger chance that you might accidentally match against someone else who happened to have their photo taken in the same pose, lighting conditions, and facial expression as your test photo, and it's very tough for algorithms to discount those confounding factors.
As for other applications of face recognition, let me describe one broad area that I think is pretty exciting, rather than a bunch of specific instances. One of the big shifts in user interface design is going to be "personalization" (to varying degrees). If a program or device or robot can recognize who you are, it can pro-actively change its settings/behavior/performance/etc. to better suit you. A very simple example is the new Kinect, which does some sort of recognition to load your saved profile/controller preferences. But that's just the tip of the iceberg.
And then of course there's personal photo collection management. If your photo organizer knew who everyone was in all your photos, it would be immensely useful in a number of ways. For starters, you could easily search for "photos of me & and my sister" or "my family", etc. You could also do more advanced analysis to start to get at more subtle things. For example, "my high school debate trip" might not immediately seem like it's related to face recognition, but in fact recognition might get you 90% of the way there.
Finally, if you've made it this far, I might as well plug my recent paper, "Photo Recall" [1], which looks at how to do advanced searches on your photo collection. Our system doesn't currently handle faces, but if you look at the kinds of queries we can do, it should become clear how they might extend with faces.
[1] http://homes.cs.washington.edu/~neeraj/projects/photo-recall...
At their talk at CVPR, they made clear that the biggest factor in getting their numbers was having tons of training data and a high-capacity learning model. The paper even shows that if you replace their 3d alignment with a standard 2d affine alignment (something people were doing 15 years ago), you lose only 1.5% on LFW (table 2).
The methods are quite different, but I think at root, it's just more being able to take advantage of much more training data.
The 97.53% "human performance" number comes from my paper from a few years ago [0]. We initially thought that there's no way we would be able to do that well on this task (manually) either, but then we tried it and our scores were much higher (very close to 100%). So the verification task on LFW is actually not that hard, although in part it's because the dataset consists of public figures and therefore you can recognize many of the people outright if you've seen them before. (We tried to measure this in our human studies but found it to be inconclusive.)
3% is not the right way to look at error rates, btw. See [1] for why. But also, face verification is more of a low-level building block for recognition, because most real-world applications don't directly care about saying whether two faces are the same person or not ("verification"), but "who is this person?" ("recognition"). A simple way to build recognition from verification is to compare a test face against all faces in the database, and then take the one that gets ranked highest. In that (simplistic) scenario, your recognition rate scales (very roughly) as acc^lg(N), where N is the number of people in your database and acc is the verification accuracy. So if, e.g., you had 100,000 people to recognize from and your verification accuracy was 97.53%, you might have a recognition accuracy of ~66% (very roughly!), which is not great.
Finally, there are already lots of applications to face recognition other than surveillance, and many more waiting to be discovered.
[0] http://homes.cs.washington.edu/~neeraj/projects/faceverifica...
As for "outperforming humans", a more accurate statement might be, "this algorithm outperforms (for this simplistic task) one experiment done with a limited set of humans on this one particular dataset which has been in the community for 10 years now and is thus highly gameable."
But I realize that's a lot less pithy.
In particular, this dataset is nearing saturation, and whenever that happens, differences in the accuracy numbers often don't mean much. So for example with Facebook's number at 97.53% and this paper's at 98.52%, you're talking about the difference between getting 148 pairs of faces wrong vs 89 pairs wrong. In practical terms, as a researcher working with a dataset like this, you very quickly learn to focus on just the ones your algorithm gets wrong, and it's impossible to not subconsciously try to optimize for getting those few cases correct, even if those techniques wouldn't actually help in the general case.
(h/t to Jay Rosen on twitter: https://twitter.com/jayrosen_nyu )
Frederic Back (the creator of the film), was present as well, though not in great health. When he was brought on stage (I think after the screening), Lasseter had tears in his eyes.
Sadly, Back died at the end of last year.
We spent the whole day in silence, walking through his forest. It was in three sections, and measured 11 kilometres long and 3 kilometres at its widest. When I reminded myself that all this was the work of the hand and soul of one man, with no mechanical help, it seemed to me that men could be as effective as God in tasks other than destruction.
sum = bottom right - top right - bottom left + top left
There's a lot of other knowledge/expertise/intuition that's required to make working implementations. There have been some deep learning tutorials at recent conferences that might be more in-depth. (See my previous comment [1] for details.)
Another good way to learn is to look at some open source implementations, such as caffe from berkeley [2] or overfeat from NYU [3].
In addition to showing how to choose architectures or set params, they also have tricks for speeding things up. This is actually very important, as they can make orders of magnitude difference (training in hours vs days).
[1] https://news.ycombinator.com/item?id=7742192
If you click through, you'll see that most of them have links to slides, and some even have video coverage and/or links to software as well. They're also usually presented by experts in that area.
Finally, note that even if the most recent conferences don't have relevant tutorials for what you're looking for, you can still often find good material by going back further in time (e.g., tutorials presented 2 or more years ago). Although the state of the art may have advanced since then, the fundamentals usually don't change very frequently.
Remember that a real person spent time & energy working on writing and releasing this app. You likely wouldn't say your last sentence to this person in real life, so please refrain from writing it here.