By voting no, you just condemned 900,000 people in Group A to eventually die, simply because you wanted to prevent them from becoming the ruling elite.
Stated differently, you had the power to prevent 900,000 deaths (and in fact many more than that, moving forward), and you chose not to because of ideology.
Surely it is better for people to not die - to not cease to exist - even if that means they rise to the top of the social hierarchy?