I always wondered how much of that effect was due to the smoothing out of irregularities in the skin that averaging produces. Reading
http://www.uni-regensburg.de/Fakultaeten/phil_Fak_II/Psychol..., I get the impression that that is a main cause of the effect. I base that on the findings that
"the more attractive the original faces, the more attractive the resulting compound face (r=0.75 for female and r=0.68 for male faces). Thus, not simply the number, but also the attractiveness of the original faces influences the average attractiveness rating of compound faces. This result is in contrast to the 'attractiveness is averageness' hypothesis."
and
"Rather, it seems that not the proportions but the surface characteristics (e.g. skin texture) of a face decide wether it is regarded attractive or not. In terms of facial proportions, this means that the 'attractiveness is averageness' hypothesis is clearly falsified."
Edit: of course, further research is needed. For example, the subjects doing the ratings all did their shoppings in Regensburg. That's way better than the typical selection of around-20-years-old psychology students (at least 80% female) of late 20th century Western European upbringing, but it will have had an effect on what constitutes beauty.