Because we live in a world that is not homogeneous and are all enriched as a result. Ending up as one's own version of Howard Hughes, locked in a room with a gun and a pile of money, is unappealing to me.
And just because there are some people you don't want to associate with doesn't mean you don't want to associate with people - just that after compulsory schooling I don't want to talk with/spend time with about 80% of the world. That leaves a bit over a billion, so it doesn't matter.