I don't really buy this study. If humans can't get a better accuracy than 55%, I'm convinced this vector is leaking some obvious (maybe high-frequency) information that doesn't have anything to do with the face. E.g location of the person.
The only relevant 55% that I can find is:
> Allport and Kramer (1946) randomly presented 20 yearbook photographs of Jews and Non-Jews to 223 undergraduate students for 15 s each and asked them to categorize the person in each photograph as Jewish or non-Jewish, or to pass on the trial by indicating a lack of knowledge. The reported median identification for the sample was slightly above chance (55.5%; Allport & Kramer, 1946). Moreover, they found that highly prejudiced people were more accurate at distinguishing Jews from non-Jews
So it's not the same set of photos and not the same question.
1. The data:
"We used a sample of 1,085,795 participants from three countries (the U.S., the UK, and Canada; see Table 1) and their self-reported political orientation, age, and gender. Their facial images (one per person) were obtained from their profiles on Facebook or a popular dating website... Facial images were processed using Face++37 to detect faces. Images were cropped around the face-box provided by Face++ (red frame on Fig. 1) and resized to 224 × 224 pixels."
2. The benchmarks:
"For example, when asked to distinguish between two faces—one conservative and one liberal—people are correct about 55% of the time."
3. The controls:
"What would an algorithm’s accuracy be when distinguishing between faces of people of the same age, gender, and ethnicity? To answer this question, classification accuracies were recomputed using only face pairs of the same age, gender, and ethnicity."
A. A complaint:
Geography and income are two powerful conditioners. These can leak in so many ways: uncropped background (geography), image color and quality (income), eyeglass shape (geography and income). This study really needs more controls. Geography and income would be a nice start.
> Their facial images (one per person) were obtained from their profiles on Facebook or a popular dating website
so of course the first thing to comes to mind is "how good of a predictor is just knowing which of those two sites the image came from?"
But then the data wouldn't represent the natural world: nature as it is.
Raw data is the correct thing to use, because it's what a hypothetical other person would also use if you ran the same experiment yourself.
The problem was, that with the ML, they ended up building a ruler classifier, because most of the pictures with skin cancer happened to also have a ruler in them to measure the size.
That way we can see how much of the performance is from magic AI pixie dust, and how much is from basic 19th century statistics.
Every time I read a paper like this, I have this Margaret Mitchell talk [1] in the back of my mind.
You think teachers are underpaid? Oh obviously you must be a pro-abortion, $15 minimum wage supporting, transgender-rights activist.
What's that you say, Christian bakers should be allowed to refuse to bake a cake with a pro-gay message on it? Oh, you must be a gun-toting, pro-life, anti-immigratnt Trump fanatic.
This kind of sorting people into simple binary categories, and giving them a "shopping bag" full of opinions they're supposed to hold helps nobody.
I'm not sure how this was relevant in anyway to your comment, but I just kinda jumped on my soapbox there.