You're not bearing any burden, as there are a lot of schools who aren't Yale that will accept you. This isn't the case for those who are benefiting from Affirmative Action.

For some, Yale is their only shot, due to the inherently racist nature of American society. Those people should get priority over people for whom this isn't the case.

If you want to be mad at someone, be mad at the racists in American society that make these programs necessary. If it weren't for the objective fact that many minority groups in the US suffer on nearly every level when compared to others, these programs wouldn't be needed, but they do so they are.