Most foreign students aren't white. Letting them stay in America doesn't play well in states that have a deficit of educated people and a surplus of electoral college votes.
That said... there is a historical record of people complaining about Irish, Polish, Italian, Greek, and Spanish immigrants. Essentially every wave of immigration has been met with hostility.
So it seems the whole they are not X like us, is just an excuse.