I don't think it's a stretch to say that life is harder (on average) for minorities.
Ignoring the role of racism and sexism to try to put all the blame on socio-economic background is a gross oversimplification. The NYtimes had a really nice infographic [1] showing the racial disparities in class mobility. tl;dr - white folks who are born rich tend to stay rich more often than black folks, and those who are born poor are more likely to move upwards.
[1]: https://www.nytimes.com/interactive/2018/03/19/upshot/race-c...