I don't think you're seeing the whole picture if you think only about physical force.
People who are well off have more opportunities to start other ventures that make them even more well-off, and if they're especially powerful, they can use that influence to further their benefit. I don't think that's particularly controversial.