Right, but people say things like 'all the speakers at this conference are white men - that's not diverse enough'.
If you are determining diversity by sex and skin color, I'm saying that you are making judgements about people's background, tastes, culture, etc that are racist and sexist.