If an opposing viewpoint backed by facts and evidence make you feel alienated, then the problem is not with the person who presented the facts. Affirmative action by definition discriminates in favor of women and underrepresented racial minorities, and this is simply a fact - the very definition of affirmative action itself.
I'm a huge proponent of diversity in the sense of embracing people with different perspectives and ideas. Deciding that you need X% of females and Y% of colored people does not make you a proponent of diversity, except in the shallow sense of wanting some set of racial representation. The strongest advocates of this form of racial diversity in my experience tend to actually be the most hostile towards diversity of opinions - like all the people deeming this memo sexist who fail to bring up any coherent argument backing up their claims (since most of them never read the document).
And one of the points of the memo was that we need to de-moralize diversity and allow people to have an honest discussion over it (which he ironically got fired over).
Diversity of thought > Diversity of appearance