World's First Live-Video Augmented-Reality Mustache App For The iPhone

After months of optimizing I was finally able to get face detection to run in under 100ms on the iPhone 4.

http://livestache.com

I thought I would share some details on how this was possible. Using the Time Profiler in Apple's "Instruments" I measured the runtime of my face detector.

Initially I used OpenCV 2.1 compiled for the iPhone as described here: http://niw.at/articles/2009/03/14/using-opencv-on-iphone/en

However my first attempt yielded fairly poor results: it took 1500 to 6000 ms for face detection to complete on a 640x480px source image.

I then spent a few days getting OpenCV 2.2 to compile, mainly because it has some nice garbage collection features and also a new face detector (the LBP detector).

The first thing I tried fiddling with which algorithm was used. I found that the LBP Face Detector was the fastest, however it wouldn't work on any of my asian friends. Perhaps the dataset used to generate OpenCV's lbpcascade_frontalface.xml file was limited? The available Haar Cascades however had no problem detecting asian faces or faces with very dark skin (definitely want to avoid an HP webcam fiasco), and among those I found haarcascade_frontalface_alt2.xml ended up being the fastest.

The next obvious thing to try was shrinking the image. Scaling it down by a factor of 5.5 before feeding it into the face detector seemed to be the sweet spot for this application. I used CoreGraphics to simultaneously convert the image to greyscale and shrink it, which I believe offloaded some of that to the GPU and I know gave a huge speed boost.

There are some performance parameters for the Haar Cascade that gave some speed boost as well. You get to specify a window size (the smallest face it will recognize), which is a compromise between speed (larger window size) and not requiring the camera be ten inches from the subject's face (smaller window size). I ended up going with a 20px x 20px window, which may seem small but remember the 5.5 image scaling factor, so really I'm requiring the face take up about a third of the 640x480px image. I set the minimum neighbors to 4, the haar window scaling to 1.2, and set the CV_HAAR_FIND_BIGGEST_OBJECT and CV_HAAR_DO_ROUGH_SEARCH flags in the parameters I passed to the face detector.

The largest single boost however came in the way I converted the image formats. As this was my first iPhone app I was startled at the number of image formats the different APIs used. There are CGImages, CVPixelBuffers, UIImages, CMSampleBuffers, and more, and then OpenCV has its own format of course. There are many code snippets out there on how to convert between the various formats, but here's where I would get sneaky.

As I was converting from a CVPixelBuffer (this is what the AVFoundation camera api returns) to CGImage, I simultaneously cropped the photo to the area of interest. Then I found that I could simultaneously shrink, rotate, and grayscale-ify this image using various CGContext manipulations, and finally I pointed an OpenCV "Mat" buffer directly into the CGImage's data buffer, thus no actual conversion was necessary. The data was only copied two times (instead of five or six as I was originally doing) and the GPU seemed to be doing some of the heavy lifting because I was able to get this to run in about 400ms.

The final speed boost that got the face detection down to about 50-100ms was using OpenCV's "region of interest" system, and setting the Region Of Interest in the image to the general area (plus or minus 50 pixels) where the last face was found (if there was one). Thus the first time it finds a face, it scans the whole image (takes 400ms) and subsequent scans look at a much smaller area, and run at a speedy 50ms-100ms.

In LiveStache you'll notice the mustache updates location 3-5 times per second (on the iPhone 4 at least). So why not more like 10 times per second if we're getting 100ms face detection times? Because it takes processor time to draw the mustache, keep the video layer running, and keep the sound running. Those add a little overhead and bring the speed down to a still-healthy 200-300ms. I suspect on the iPhone 5 this will drop significantly, and we will be able to achieve real-time (10fps or so) face detection.

I hope this is helpful information for those of you with cool ideas that center around Face Detection.

Ian Charnas ian.charnas@gmail.com