It's just a figure of speech. Obviously, citizens have the right to vote for whatever immigration policy they would like. Maybe I should have said "ethically, you should not deny others a benefit that was granted to you", which is what I meant.