What if the user had entered the contact info manually? Still an invasion of privacy?

In the extreme, this position gets really absurd. I never consented to allow gmail to store my email address in my friends' contact lists. At some point, "your" data becomes your friend's data and it's no longer yours to control.

Same thing with Viber and other apps. Some of friends use their services and now they have my contact information and can build up a shadow profile on me. I consented to none of that. While they may deny that's their intent it doesn't change the fact they have all that data.