The disagreement is rooted in the question of whether economic power is relevant for an individual's liberty or not. There are those who are fine with curtailing the liberty of those who don't have economic power (or the flip side: who claim that extreme wealth does not increase a person's freedom)[0], and there are those who see inequal access to economic power as a problem that needs to be fixed in order to improve overall individual liberty.
[0] If you hold that view but have ever considered the term "fuck you money" to be a valid meme, then I've got bad news about the logical consistency of your worldview.
IOW, they argued that there cannot be a just system where everyone is truly equal when some of the people have the power to apply force (government) that, when applied by those not chosen to have such powers (citizens), is "illegal." (E.g., taxes, war, imprisonment, drafting of laws, etc.)
The restriction of rights to non-whites was a policy created and enforced by governments. Also enforced by governments were the Jim Crow and other eugenics-based laws (including forced sterilizations, which disproportionately affected racial minorities). The 13th amendment was written to only ban private ownership of slaves - governments can and do still own slaves (prisoners perform forced labor that profits the state and other entities, not the harmed parties).
Access to opportunity is most heavily curtailed by governments, who make massive laws that heavily increase the cost to open new or even maintain existing smaller businesses. In Silicon Valley, we see an extreme version of a common problem created by governments where the governments act to restrict the building of housing, resulting in housing only being available to the extremely wealthy.