Of course privacy matters, even if you are doing nothing wrong. Without it, we'd continually have to explain our actions and prove our innocence to people who don't understand our situation.
And that's a lot of work. And they really aren't motivated to understand.
And its a lot of work trying to understand everyone else's situation. I certainly don't want to.
So nature gave us a need for privacy and the means to respect the privacy of others. And even the sense to often pretend that we don't see many things.