It's just a figure of speech. Obviously, citizens have the right to vote for whatever immigration policy they would like. Maybe I should have said "ethically, you should not deny others a benefit that was granted to you", which is what I meant.
Even if doing so is a net negative? Remember when Tunisian and Saudi nationals didn't drive cars through crowds at Christmas markets in Germany?