Phone numbers are not private. The identifiers we use to connect to one another are not private. By design.
But I still don't want the entire world knowing that he's in my contact list.
As you already pointed out there's a gigantic downside which is your privacy. I always keep hearing the occasional person that says that they are not on Facebook (for privacy reasons) but uses WhatsApp. I later congratulated them for signing up to Facebook and they also allowed WhatApp to upload their entire address book to find or search for anyone's number on Facebook who is not on WhatsApp!
Wire, Signal, and Telegram do the same thing but are just as bad for privacy and are disqualified.