Not so. They split the data between "bearded faces" and "non-bearded faces" and compute the vector from one to the other, and use that to alter a given face. (It reminds me of the typical word2vec example of man - woman + queen = king.)
See their code snippet halfway down their page, or "Semantic Manipulation" on page 8 of their linked paper.