If your data set was compiled by a single person, well, easy enough.
If your data set was compiled by thousands of mechanical turk workers, well, you got a lot of people to blame a little bit. As it goes, "everybody is a little racist sometimes..." and apparently that shows on a big enough data set.