One of the bigger ways this plays out as opposed to your example is: Tons of property is locked up as short term vacation rentals that are only used a tiny percent of the time and only by the rich. There's a spectrum of how rich but we know the bottom 50% almost never use them for example
Similarly the amount of resources locked up in industries that 99% of the time only cater to the very rich is quite a lot and more importantly the trajectory is going more and more that direction.
You could have a world where the work is done mostly by robots and a few million rich people use the world as their playground and then what happens to everyone else?