People don't have to be "damaged" by something to object to it. How do human rights abuses in third world countries "damage" you, exactly? Just because there isn't a direct link doesn't mean that there's nothing bad going on.
I myself don't care that this data was used to help Donald Trump ascend to the presidency -- besides, I bet that Hilary's campaign did similar sketchy things with private data. The issue is that our private data is being abused by a company that misled us about data use -- and for folks who don't even have a Facebook account, whose information has been harvested via shadow profiles sourced from our less-privacy-conscious friends, data compilation occurred entirely without consent.
This reeks of whataboutism to me. Why are you trying to imply that we shouldn't be concerned about privacy?