I don't even think robotics are the primary thing to be concerned with. There would be a whole slew of additional hardware problems to solve.... There is a TON that can be done outside the physical world that is probably far more damaging
Introducing millions of robots into society is something for which specific safeguards should be built. Costs related to developing such safety mechanisms seem small compared to the safety they can provide.
There will be many concerns relating to AI and robotics that should be addressed by a multitude of different safeguards.