Any freedom is some compromise. If a society has two people or more, then either one person is allowed to, say, enslave the other, in which case the society isn't totally free, or not, in which case the society also isn't totally free. So there is no such thing as absolute freedom, and whenever we say freedom we actually mean some point on a spectrum. We could argue over what that reasonable point is, but absolute freedom is something that can't exist. So instead of speaking in absolutes, let's acknowledge that we're arguing over a favorite compromise.
It's not a moral principle. It's an explanation of where we find ourselves. This doesn't mean we shouldn't keep striving for a just society, but we don't live in a just society.
Or maybe we can develop really advance VR so that make possible for everyone to live in their own ideal socity.
If everybody simply stuck to treating others how they would want to be treated, we'd live in a much better world already, even if it wasn't "perfect", and even if there were disagreements, and it all still always subject to constant learning and reflection.
Our main problems don't stem from out confusion about what we think is right (and by "we" I mean each of us as the individuals that actually exist, not as a collective abstraction), but from wanting what we think is right for ourselves, while having double standards for others, and rationalizations for those.
[0] I know there are such policemen, I extrapolate to expect there are likeminded judges too.