I don't know what your argument about tallness is supposed to mean. Fairness has nothing to do with people being equal in physical attributes, however it does have to do with being equal in front of the law and how to influence law and the ability to improve one's economic situation (or the chance to loose it).

That is what is being eroded by the wealth inequality. The wealthy are significantly insulated from the law, are shaping laws that further reduce their chance of loosing wealth, increasing their wealth in the process and reducing everyone else's economic opportunity.

For a society this creates an unstable situation, if all economic (and political) power sits in the hands of only a few that creates unrest often leading to a violent overturn of the situation (or trying to counteract via totalitarian measures).

As a side note, you say you're content with not being wealthy, but how far would this content stretch, would you be content if you are so poor that you'd struggle to feed your kids?