> it follows that it'll be able to replace all physical labor with robotics fairly quickly
This is not clear at all. Even the most expensive robotic/arms hands perfectly controlled fail to do thing as quickly and easily as human limbs, not to mention the chaotic environment that is the real world. It's clear that it can happen, but the timeline is not quick.