The problem with just having the employee and the employer come to a contract, is that the negotiation position is not on equal terms. And since the economy is here to serve us; not the other way around; we democratically elected this policy and have preferred it for almost half a century.
Your notion of rights, esspecially economic rights, as somehow inherent to reality as laws of nature, seems awfully short sighted. My neigbour is not free to have himself exploited, as it will affect my negotiation position as well.
And i honestly dont understand why so many Americans keep focusing on their rights to be exploited and abused in an economic sense. Of all inalienable rights i can think off, like autonomy of your own body (drugs? sex?) or freedom of speech (swearing?), that do not legally, socially or culturally apply to americans, the narrative limits itself to guns and slaves (exploitable workforce). The only two situations where it obviously does affect others, aand therefor the right can not be inherent.
I would not trade my rights for yours.