More broadly, laws aren't worth the paper they're written on if you don't have a police and armed forces willing and able to uphold them.
I understand what your point is; I've talked to several people who voice the same opinion when discussing my country's dark past, but trust me, when the bullies get to power, you can't shout "it's a human right!" back at them. You either have to fight back, hide, or run away.
Simple example: Religious freedom/freedom of speech and gay rights. So should the Christian baker be _forced_ to make a cake with words explicitly against his religious beliefs? Which human rights come first here?
http://criticallegalthinking.com/2019/07/12/hannah-arendt-ri...
If you're employed at a desk job, you've traded much of your human right to be wherever the heck you want between the hours of 9 and 5, Monday through Friday, in exchange for a salary.
Soviet bloc countries didn't vote for the Universal Declaration of Human Rights because of the right of citizens to leave their countries. And let's not get into the human rights of Uyghurs and the worldwide approval/disapproval to China's re-education camps.
The salient point being that indifference at scale is worse than individuals with explicit bias.