You're making an argument based on popular vote, but black people and white people wouldn't be able to intermarry either if our system was entirely based on the will of the masses.
Not all states use the Electoral College (4 to 6 don't), more and more left leaning states want pure majority.
http://en.wikipedia.org/wiki/Electoral_College_%28United_Sta...
Also, each presidential term is limited to 4 years.