That wasn't the claim. The claim here is that we should scrub certain faces from the dataset in order to change the dataset in a certain favorable way.
That way we can see how much of the performance is from magic AI pixie dust, and how much is from basic 19th century statistics.
Every time I read a paper like this, I have this Margaret Mitchell talk [1] in the back of my mind.