There's no point in having either of those if we collectively do not enforce it.
Your position would allow to introduce arbitrary limits on freedom because you can always show a direct or indirect cost caused by having freedom. (it would certainly be cheaper to have everyone living in safe and orderly labour camps, wouldn't it?)
Regarding the U.S. Constitution: I'm in the EU but from my basic understanding the Constitution is a list of supreme laws limiting the governments power to interfere with citizens freedoms and not the other way around. Compiling a list of citizens freedoms would be senseless anyway as it would quickly approach an infinite list. Since I have not heard about a Constitutional law that allows the government to displace people internally it follows that the government does not have this power.