> there being a whole underclass of workers who can't just freely move between jobs,

I think you hit the nail on the head here. The job switching process for foreigners should be easier.