Yes, the transition is unlikely to be linear and without conflict, if this was ever possible. But I am sure that some would be happy to control armies of bots and very few humans.
And I agree that it would pose many unforeseen challenges.
This is why the transition is the interesting part, not the sci-fi end game with a world populated with billions of robots doing everything.