I think you are really underselling how awful life has been in the past.
Just look at things like child mortality rates.... historically, nearly HALF of all people died during childhood. That is insane. The global rate is now about 1/10 of that.
Similar changes have been made to starvation and malnourishment, with today's rates so much lower than historically.
This would not be possible without the improvements in efficiency over the years. Because we need so few people to grow enough food to feed everyone, we can spend the resources on medical research and advancements that have allowed our health outcomes to improve so much.
Are the gains of an efficient society unequally distributed? Of course. But let's solve the inequality problem by sharing the gains, not destroying the gains.
Yes, if we got rid of our efficiency gains, we would be more equal... because everyone would be much poorer.
We would still have people exploiting others, though. That happened way before we were efficient.