This sounds like the same point that people bring up about privacy, i.e. "if you have nothing to hide you shouldn't be afraid". While the point is valid the issues are more based in the principal of the matter.
So then since you can’t always have it your way when you’re living in a society, isn’t it that the only other option to have things your way is for you to live alone, all consequences entailing? Therefore, isn’t the argument for the full exercise of individual freedom an argument for the dissolution of society and ergo government? Because this topic seems to me about the full realization of individual freedoms and not making any compromises at all.
An important detail here. Technically you are kind of correct, but sadly there are plenty of examples of people who should have nothing to hide who have rightly had reasons to fear.
I.e. many people have reasons to fear without having anything to be ashamed of.