You don't need to suspect it. You can read the social science literature and have some confidence that hard-working researchers have confirmed your suspicion.

Here's a good place to start: Income Inequality and Social Dysfunction by Wilkerson and Pickett.