The founding fathers were all privileged white men.
I think that he has a point in saying that if you have an heterogeneous society, it's not wise to put all the power in an homogeneous group.
I like affirmative action, but I don't think it is a viable principle for determining political rights, which should be universal and colorblind rather than being denied or granted based on race or a distaste for homogeneity.
The idea that white people should not have power because of their inherent moral depravity is definitely a racist idea.