When it starts affecting people who vote, they'll start to push back and you'll get either Keynesian economic stimulus (we'll pay you to dig a hole and fill it back in again), or regulation that makes automation more difficult (e.g. French-style job protections).

Put in a policy saying that anyone whose job gets eliminated or reduced as the result of automation is eligible for half-pay and full pension benefits until they have a new job at an equivalent salary and suddenly the business case for robots wouldn't be nearly so strong. And people will be pushing for stuff like that if they feel that their jobs, rather than some Polish guy's, is at risk.

Migrants, by and large, don't vote, so you can automate their jobs away without incurring regulatory risk to yourself, if you are a large organization. It's the safe place to start.