> Are my rights infringed if the infringement has no impact on any of my other rights or my life in general
You're assuming no impact without evidence. Let's think about that first. Even assuming no G-men in black helicopters descend on you at any point in your life, what results from the government collecting all of this information?
The most obvious problem is, what happens if the government uses the same level of competence to keep your secrets secret as they do at anything else? What happens when some tool walks out of the NSA with a backpack full of hard drives comprising the metadata of everyone in the US for the past decade, and posts it on the internet?
The results of that are entirely predictable. Every vulnerable population whatsoever will be exposed to their antagonists. Crime bosses will be able to find informants and undercover law enforcement officers. Terrorists and criminals will be able to identify vulnerable targets. Industrial espionage and blackmail will proceed on a massive scale. Abusive spouses will find their exes. None of that reasonably classifies as "no harm no foul."
But OK, that hasn't happened yet, suppose we give them the benefit of the doubt. Uncle Sam is going to fill the server room with Marines until the end of days and keep everybody's secrets from ever leaking onto the internets. But still there is the government. If you know the government is recording your movements, are you going to be willing to visit a Communist rally to broaden your horizons or satisfy your intellectual curiosity? What about an abortion clinic, or a psychiatrist? What aren't husbands and wives going to be willing to say to each other for fear that someone could be listening? What essays aren't going to be published if their authors don't believe in their ability to remain anonymous against pervasive surveillance?
The problem with "harmless" surveillance is that it creates fear, and fear is a harm.